How to Interact With and Debug a gRPC Server

Photo by Markus Spiske on Unsplash

Let’s learn how to interact with and debug a gRPC server. An issue with gRPC and protobuf, if you are used to REST-like with JSON, is that it can be a bit more complicated to debug and troubleshoot, and the tools are generally more specialized. This article will serve to introduce you to the most useful tools for quickly working with this ecosystem.

CLI tools

Here are some useful tools that you can use to interact with a running gRPC server on your local machine:

• grpcnode — CLI tool for quickly making servers and client, dynamically, in JavaScript.
• grpcc — REPL gRPC command-line client.
• grpc_cli — gRPC CLI tool.
• Evans — Expressive universal gRPC (CLI) client.
• grpcurl — Like cURL, but for gRPC: Command-line tool for interacting with gRPC servers.
• danby — A gRPC proxy for the browser.
• docker-protoc — Dockerized protoc, grpc-gateway, and grpc_cli commands bundled with Google API libraries.
• prototool Useful “Swiss Army Knife” for processing proto files.

An example

Install grpcnode.

npm i -g grpcnode

You can start a quick “Star Friends” server, from the example repo:

git clone https://github.com/backstopmedia/gRPC-book-example.git
 cd gRPC-book-example/examples/nodejs
 npm i
 npm start

grpcnode server

You can use it to get a list of available RPCs, based on the proto:

grpcnode client ls -I ../../proto sfapi.proto

grpcnode ls

You can run the GetPerson RPC by JSON-encoding your request:

grpcnode client run -I ../../proto sfapi.proto -c ‘/sfapi.v1.Starfriends/GetPerson({“id”:”SyAbJp35ViM”})’

There is also a HTTP/JSON gateway running (via grpc-dynamic-gateway, so you can use your favorite way of messing with JSON-servers (curl, postman, etc.)

grpcnode ls

Environment variables

You can troubleshoot and change how C gRPC works by setting a few environment variables. There is a more complete reference of these here but the most important are GRPC_TRACE and GRPC_VERBOSITY.

GRPC_TRACE

This is a comma-separated list of tracers that provide additional insight into how gRPC C core is processing requests, via debug-logs.

all can additionally be used to turn all traces on. Individual traces can be disabled by prefixing them with -.

refcount will turn on all of the tracers for refcount debugging.

if list_tracers is present, then all of the available tracers will be printed when the program starts up.

Example:

export GRPC_TRACE=all,-pending_tags

This means “put a trace on all available operations except pending_tags”. There are quite a few traces you can turn on and off to debug what part of the request is failing, or just see more detail about what is happening. These will work for any library that uses the C core (for example ruby, node, python.)

Available tracers include:

• api — traces api calls to the C core
• bdp_estimator — traces behavior of bdp estimation logic
• call_combiner — traces call combiner state
• call_error — traces the possible errors contributing to final call status
• channel — traces operations on the C core channel stack
• client_channel — traces client channel activity, including resolve and load balancing policy interaction
• combiner — traces combiner lock state
• compression — traces compression operations
• connectivity_state — traces connectivity state changes to channels
• channel_stack_builder — traces information about channel stacks being built
• executor — traces grpc’s internal thread pool (‘the executor’)
• glb — traces the grpclb load balancer
• http — traces state in the http2 transport engine
• http2_stream_state — traces all http2 stream state mutations.
• http1 — traces HTTP/1.x operations performed by gRPC
• inproc — traces the in-process transport
• flowctl — traces http2 flow control
• op_failure — traces error information when failure is pushed onto a completion queue
• pick_first — traces the pick first load balancing policy
• plugin_credentials — traces plugin credentials
• pollable_refcount — traces reference counting of ‘pollable’ objects (only in DEBUG)
• resource_quota — trace resource quota objects internals
• round_robin — traces the round_robin load balancing policy
• queue_pluck
• queue_timeout
• server_channel — lightweight trace of significant server channel events
• secure_endpoint — traces bytes flowing through encrypted channels
• timer — timers (alarms) in the grpc internals
• timer_check — more detailed trace of timer logic in grpc internals
• transport_security — traces metadata about secure channel establishment
• tcp — traces bytes in and out of a channel
• tsi — traces tsi transport security

The following tracers will only run in binaries built in DEBUG mode. This is accomplished by invoking CONFIG=dbg make <target>.

• alarm_refcount — refcounting traces for grpc_alarm structure
• metadata — tracks creation and mutation of metadata
• closure — tracks closure creation, scheduling, and completion
• pending_tags — traces still-in-progress tags on completion queues
• polling — traces the selected polling engine
• polling_api — traces the api calls to polling engine
• queue_refcount
• error_refcount
• stream_refcount
• workqueue_refcount
• fd_refcount
• cq_refcount
• auth_context_refcount
• security_connector_refcount
• resolver_refcount
• lb_policy_refcount
• chttp2_refcount

GRPC_VERBOSITY

Default gRPC logging verbosity — one of:

• DEBUG — log all gRPC messages
• INFO — log INFO and ERROR message
• ERROR — log only errors

Decompiling protobuf messages

You have a few options if you don’t have the original proto IDL, but want to build it or just guess it from a binary protobuf message.

If you can at all help it (get your hands on the proto IDL for the data) it’s recommended that you don’t parse binary messages without the proto. It’s unreliable, as many fields can share the same types (e.g., it’s impossible to tell the difference between number-types & enums.)

• If you just need to see the format, you can use protoc — decode_raw to guess at the types.
• You can use rawproto to extract the same sort of data as JSON or proto, which is a useful start, if you tune the output a bit
• You can parse it by hand. This is pretty tricky, and not at all recommended as it’s the most error-prone, and one of the other options should get you where you need to be, but there is documentation about how the messages are encoded here.

Summary

Now, you should be able to quickly run CLI commands to interact with your running gRPC server, at a high-level. You should be able to troubleshoot your server, and see what the problem is, at a very low-level.

Credits

This post was excerpted from the book titled, “What is gRPC.” If you’re interested in learning several more advanced topics for making the most of gRPC, please check out “Practical gRPC” by Joshua Humphries, David Konsumer, David Muto, Robert Ross and Carles Sistare, a how-to guide published by Bleeding Edge Press.

Humble Bundle Special Offer

Bleeding Edge Press has partnered with Humble Bundle to offer up to $362 worth of DRM-free Open Source e-books, including Practical gRPC, all in multiple formats for all devices, for as little as $1. A portion of the funds go to support @GirlsWhoCode

For a limited time, you can find the offer here.