Building a realtime dashboard with ReactJS, Go, gRPC, and Envoy.

Niraj Fonseka
Jun 19, 2020 · 11 min read

Before we start, you can find all the code related to this article in this repository.

In this article, I will be taking you through the process of building a real-time dashboard using Go, ReactJS, and gRPC. I’m writing this article with the assumption that you have a basic understanding of the technologies mentioned above. But I will make sure to explain and provide documentation if you are interested in learning more about a certain topic. So what is gRPC ? gRPC is a RPC ( Remote procedure call ) framework that was built and open-sourced by Google. In simple terms, an RPC means invoking a procedure in a remote server. One cool thing about this is that the developer doesn’t need to explicitly code the details of the network interaction. It’s all handled by the framework instead. As you will see in this article, invoking an RPC feels like you are just calling a function in another server. And also this can be done even if the server and the client are written in different languages.

We will be building a Go server that uses gRPC to communicate with our client, which is a javascript app. gPRC allows you to communicate in multiple methods.

Unary RPC

Client sends a request and gets a response back from the server ( this is similar to REST )

Server-side streaming

Client sends a request and gets a stream of messages from the server

Client-side streaming

The opposite of server-side streaming. Client sends a stream of messages to the server

Bidirectional streaming

Server and Client send streams of messages to each other. How you process these messages are based on how your application is structured. But these two streams are independent of one another so you have more flexibility when it comes to deciding on how to handle these streams.

For this project, our goal is to build a dashboard that updates in real-time. Let’s assume we are building this dashboard to display some sensor data. And our server will be collecting all the sensor data for us. Therefore in this case our client should invoke a request to the server ( saying I’m ready to accept data ) and then the server should keep sending data to the client whenever the sensor value changes. So it looks like Server-side streaming is the most suitable in this situation.

Setting up the server

  • protobuf or protocol buffers is a mechanism for serializing structured data invented by Google. It tends to be smaller and faster than other popular serialization standards such as JSON and XML. But unlike JSON and XML protocol buffers can be used for defining services interfaces ( not just message interchange formats )
  • gRPC uses Protocol Buffers as it’s Interface Definition Language. Therefore we can define our service interfaces and data structures in a proto file and then use the protoc compiler to generate our Go code for us. This is also a great benefit of using gRPC because we can generate code for many languages just by using one proto file.

sensor.proto

In this proto file we have defined two messages. One for SensorRequest and another for SensorResponse. The reason why theSensorRequest has no fields is that we are not expecting to pass anything into our RPC functions. In the SensorResponse message, we have a int64 field called value. This field will carry our sensor data. In the Sensor service, we have defined two RPC functions. One for getting the temperature and one for humidity. Both of these return a stream of SensorResponse messages.

Let’s save that file in a directory called proto. And then let's create another directory called server at the same level as the protos directory. And create another directory called sensorpb inside protos directory. This is where we will store our generated Go code. So the directory structure should look like this.

-- protos 
- sensor.proto
-- server
-- sensorpb

First, let's install some dependencies. We will need the protoc compiler and the protocol compiler plugin for Go.

Install the protoc plugin: https://github.com/protocolbuffers/protobuf
Install the Go plugin :

export GO111MODULE=on # Enable module mode
go get github.com/golang/protobuf/protoc-gen-go@v1.3A

After installing the dependencies mentioned above, run this command within the protos directory to generate client and server Go code and then store them in theserver/serverpb directory.

protoc sensor.proto — go_out=plugins=grpc:./../server/sensorpb

If you examine the server/sensorpb directory you will see a new file named sensor.pb.go. Now that we have the generated code we are ready to write our server.

Server.go

Let’s breakdown this code.

  • In the main function, we will first create a Listener and tell it to run the server on port 8000
  • And then let’s create a *grpc.Server instance by calling grpc.Server()
  • Call the RegisterSensorServer function by importing the code that we generated as a package to register our sensor server. This function takes in a *grpc.Server and a type that satisfies the SensorServer interface ( you can examine the code in the generated sensor.pb.go to the function definition ).
  • Therefore we will have to define a type and then create two methods on that type called HumiditySensor and TempSensor to satisfy the SensorServer interface .
  • We have defined a type called server and created those two methods in the code block above the main function.
  • Now, let's run our app.
go run server.go

Since we don’t have real sensors for this example we will have to generate some fake data on our own. So let’s create a package called sensor that generates random data and stores them in a map datastructure.

Our new directory structure should look like this.

-- protos 
- sensor.proto
-- server
-- sensorpb
-- sensor

Let's do a quick run through of the sensor package code.

  • setHumiditySensor and setTempSensor functions generate random data and store them in a map. setHumiditySensor sets a new value every 2 seconds, setTempSensor function sets a new value every 5 seconds.
  • StartMonitoring function will run those two functions in the background as Goroutines and keeps generating data until we stop the application.
  • GetTempSensor and GetHumiditySensor functions will return the values for the humidity and temperature sensors.

Now let’s modify the server.go code

  • First, we have modified our Server type to have an instance of Sensor so we can call methods to get metrics. And we have changed TempSensor and HumiditySensor functions as well. Let's take a look at one of those functions.
  • In the TempSensor function, we access the Sensor instance and in an infinite loop, we call the GetHumiditySensor function to get the latest value. And we sleep for 2 seconds because we know the value updates every 2 seconds. Now that we have the latest value to send through the stream, we call stream.Send function and then pass in a SensorResponse with our metric assigned to the Value field.
  • In the main function, we create an instance of Sensor by calling the NewSensor function and then we call StartMoniotirng function to start generating out "metrics". Then we assign an instance of Sensor (sns variable) we created into the Sensor field in the server when we register the SensorServer .

Now that our server is complete let’s talk about creating a JavaScript client that can display our data in real-time.

Creating the js client

I’ll use create-react-app to set up our react app.

npx create-react-app js-client
  • Now just like we did previously for Go, we will need to generate the client and server code for Javascript. We can use our sensor.proto file again for this. Let's create a directory called sensorpb inside the jsclient/src directory to store our generated files. But since our client will be a browser client so we will have to use grpc-web. Before we talk about this more let's talk about a big problem we have.
    Most modern browsers don't support HTTP/2 ( yet ). Since gRPC uses HTTP/2 we need to figure out a way to make our browser client communicate with our gRPC server. grpc-web makes this somewhat possible. grpc-web allows us to use HTTP/1 with a proxy such as Envoy which helps us to translate HTTP/1 to HTTP/2 so that we can still communicate with the gRPC server.
  • Make sure you have the protoc-gen-grpc-web plugin installed → https://github.com/grpc/grpc-web installed
  • And then run this command to generate the code.
  • In the js-client/src we can find sensor_grpc_web_pb.js and sensor_pb.js files generated.
  • Let's clean out the App.js until we are just left with a plain component.

Great! Let’s stop working on the client for a bit and focus on setting up our proxy because we will need it to fetch data from our server.

Now let’s work on setting up the Envoy proxy. Again, the reason for this is because we want our js-client to talk to Envoy first using HTTP/1 so it can translate those requests to our gRPC server by translating to HTTP/2.

Setting up Envoy

  • I’ll be using Docker to get an Envoy proxy up and running.
  • We will have to define envoy.yaml with the configurations we need.
  • This is a slight modification to the example provided in the grpc-web repository. But I’ll do a quick walkthrough of the configuration file to explain the most important things.
  • Clusters are a group of upstream hosts that accept traffic. In this case, it's our gRPC server. Therefore we will have to set the hosts field to point to our server address in this case [localhost:8080](<http://localhost:8080>) .
  • Listeners can accept downstream connections. In this case, it’s our js-client
  • Filters essentially add extra features. There are many types of filters. For example, Listener filters allow you to manipulate metadata of Layer 4 connections during the initial connection phase. Network filters allow you to manipulate Layer 4 layer connections as well. HTTP filters allow you to manipulate HTTP requests and responses while operating at Layer 7.
  • Routes allow you to match virtual hosts to clusters and create traffic shifting rules. For this example, we are using a static definition of a route but these also can be defined dynamically via the route discovery service ( https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/rds )
  • So in that envoy.yaml file, we are essentially asking Envoy to run a listener on port 8000 that listens to downstream traffic ( this is the port our js-client will be pointing to ). And then direct any traffic that comes to it to the sensor_service which is our gRPC server running on port 8080.
  • Now let's create another directory called envoy in the root of the project and then store the envoy.yaml file in there. Now the directory structure will look like this.
-- protos    
- sensor.proto
-- server
-- sensorpb
-- sensor
-- js-client
-- sensorpb
-- envoy

And then in the same directory let’s create a Dockerfile to also include the envoy.yaml file that we just created.

Let’s build the Docker image now.

docker build -t grpc-medium-envoy:1.0 .

Now let’s run it.

docker run --network=host grpc-medium-envoy:1.0
  • Now that we have Envoy up and running let's get back to work on our js-client

js-client continued …

First, let’s install some dependencies.

npm install grpc-web --savenpm install google-protobuf --save

Let’s import SensorRequest and SensorResponse from sensor_pb.js and SensorClient from sensor_grpc_web_pb.js and create a variable for the client.

  • Note how the address that we provided is pointing to port 8000 , not the port where our gRPC server is running. That's because we want our js-client to talk to the Envoy proxy instead.
  • Let’s try to get our temperature value displayed first. For this section, I will assume that you have a basic understanding of React hooks. You can read about them more here ( https://reactjs.org/docs/hooks-reference.html )
const [temp, setTemp] = useState(-9999);

And then let’s create a function to read from the stream. In that function let’s first create a sensor request. And then let's create stream variable by calling theclient.tempSensorfunction. And pass the sensorRequest in.

var sensorRequest = new SensorRequest() 
var stream = client.tempSensor(sensorRequest,{})

Finally, let's call stream.on and then pass in a callback function to handle the response that we get from the stream. And inside the call back function let’s call setTemp function and pass in the value from response.getValue() which is the value that was read from the stream.

const getTemp = () => {  var sensorRequest = new SensorRequest()
var stream = client.tempSensor(sensorRequest,{})
stream.on('data', function(response){
setTemp(response.getValue())
});
};

That way we will be updating the state every time a new value is read from the stream. And then let's call the getTemp() function inside the useEffect hook.

useEffect(()=>{
getTemp()
},[]);

Note how we pass an empty array for the second argument in the useEffect hook. The reason for this is because when you pass in an empty array as the second argument, it almost works like the componentDidMount() life cycle function. Meaning it will run once when the component renders and mounts for the first time. And then when the stream.on function gets invoked it will keep updating the state every time there’s a new value. This causes the page to render a new value every time the state changes.

Now let’s start the app by running npm start . You might see a compilation error. You can get around this by adding /* eslint-disable */ at the top of sensor_grpc_web_pb.js file and the sensor_pb.js file. If you are interested you can learn about this error from here https://github.com/facebook/create-react-app/issues/7295.

sensor_pb.js

Let’s try running npm start again.

Great !! We can see that the temperature value is changing every 5 seconds as we expected. Now let’s do the same for humidity as well.

As you can see the humidity value changes every 2 seconds while the temperature changes only every 5 seconds just as we expected. The finalized App.js file will look like this.

And that’s it! If you want to make this pretty, with some CSS magic you can end up with something like this.

The temperature box becomes red when the temperature is greater than 90 or less than 30. The humidity box changes it’s color to red when the humidity is above 80.

If you made it this far, I hope you enjoyed it! Please leave a comment if you have any questions. Thank you for reading.

source

The Startup

Get smarter at building your thing. Join The Startup’s +800K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store