Getting started with Quarkus and InfluxDB to ingest sensor data from a Particle device — Part 2

Photo by Felix Mittermeier from Pexels

Welcome back to our second part of Getting started with Quarkus and InfluxDB to ingest sensor data from a Particle device! In this tutorial, we are going to explore the world of time series databases. We’ll show how to set up an InfluxDB Cloud account and how to ingest data through a Quarkus application.

ICYMI, in the first part of this tutorial series, we wrote our first controller using Quarkus. We implemented a POST endpoint to retrieve data from a Particle device through a webhook or, in case you don’t have a Particle device, we used k6 to simulate a stream of random values. If you haven’t read it, we suggest you start from there, otherwise, if you want to go straight to InfluxDB Cloud, you’re in the right place.

Getting started with Quarkus and InfluxDB to ingest sensor data from a Particle device — Part 1

So, let’s get started and remember that you can find all the code for this tutorial and the previous one on GitHub (the code for part 1 of the tutorial can be found in the branch part-1).

What is a time series

A time series is a sequence of measurements performed at successive points in time. — nature.com

A time series is a type of data that is usually characterized by a timestamp, some metadata, and measurement values. Also, time series data usually come at regular (and oftentimes frequent) intervals of time. Time series are everywhere, just think about the amount of data coming from IoT sensors, or logs generated by software applications.

Why InfluxDB?

InfluxDB is a database designed for time series storage and queries. There are multiple benefits in using InfluxDB when working with time series. Two of the major ones are:

• Fast ingestion rate: The indexing system of InfluxDB is optimized for data aggregated by time, guaranteeing high ingestion rates even when the amount of collected data grows.
• Retention policies: It doesn’t take too much time with time series to pile up a huge amount of data. Retention policies set an “expiration date” on your data making sure to drop them when they are no longer useful.

InfluxDB setup

The first thing we need to do is to create a free InfluxDB Cloud account. You can deploy an instance of InfluxDB on your dedicated server if you want, however, InfluxDB Cloud is a managed cloud service that makes it a great option to get started. For this tutorial, we’ll use the Google Cloud Platform as a provider, but any other provider will work as well.

Create a new bucket

Once registered, go to the Data section and create a new bucket. We are going to name it ParticleData and set the retention policy to 14 days.

To connect to InfluxDB from our backend we’ll need the following pieces of information:

• Bucket ID
• Auth-Token
• Organization ID
• Connection URL

Bucket ID

You can find the Bucket ID in the Data→Buckets section under the name of the bucket you just created.

Auth-Token

To generate a new Auth-Token, go to Data→Tokens→Generate. Our suggestion is to generate a Read/Write token with permissions only for the bucket you just created. Give it a meaningful name and once you save it you’ll be able to find the Auth-Token string by clicking on it.

Organization ID

To find the Organization ID instead, go to Profile→About and you should be able to see the ID on the right side of the screen under the section Common Ids.

Connection URL

Finally, the Connection URL is simply the URL that we can see when we are connected to the InfluxDB Cloud dashboard. The URL will be of this format: https://<CloudProviderZone>.<CloudProvider>.cloud2.influxdata.com. For example, we selected Google Cloud when we signed up, so our connection URL will be https://us-central1-1.gcp.cloud2.influxdata.com/.

Persisting data in Quarkus

If we go back to the Quarkus project that we started here. We will continue the project that we started in Part 1 of this tutorial series and we will extend it so that it will persist the data coming from our sensor to the InfluxDB bucket that we just created.

First, we have to update our dependencies and add the following to the POM file:

<dependency>
  <groupId>com.influxdb</groupId>
  <artifactId>influxdb-client-java</artifactId>
  <version>1.10.0</version>
</dependency>
<dependency>
  <groupId>com.influxdb</groupId>
  <artifactId>flux-dsl</artifactId>
  <version>1.10.0</version>
</dependency>
<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-hibernate-validator</artifactId>
</dependency>

• InfluxDB Client Java: the official Java client for accessing an InfulxDB service
• Flux DSL: the official Java library for programmatically build queries in the Flux language
• Hibernate Validator: library for validating the input/output of REST services

Let’s also add the following configuration details to be able to connect to InfluxDB in the application.yaml file:

influxdb:
  connectionUrl: ${INFLUXDB_CONNECTION_STRING}
  token: ${INFLUXDB_TOKEN}
  orgId: ${INFLUXDB_ORG_ID}
  data:
    bucketId: ${INFLUXDB_DATA_BUCKET_ID}
    bucketName: ${INFLUXDB_DATA_BUCKET_NAME}

Note that you should either define the value of the variables as environment variables or, directly substitute them with their value in the application.yaml file.

Now we can proceed with the creation of a POJO class that will represent our time series data.

You can see that the class is annotated with the InfluxDB’s @Measurement annotation so that once we send the data to InfluxDB they will be mapped to a specific measurement inside the chosen bucket for that connection.

Looking at the attributes of the class instead we can see that they have the @Column annotation. Such annotation will map the attribute to an Influx Field by default or, if we specify that, to a Tag. It is important to understand the difference between a Field and a Tag. Influx does not index Fields. That means that a query that performs a select on a Field will result in a scan and regular expression matching applied to our data (it will take some time before we have a result). On the other hand, a Tag will be indexed so that we will have fast query selection.

Fields and Tags are among the key concepts in InfluxDB. If you want to learn more about them check out the official Influx guide on data elements.

We are now about to write all the code needed to connect our Quarkus application to InfluxDB. First of all, we will write a custom exception that we will use for exceptions raised in our DataService.

Let’s now create a new class that will implement the IDataService interface. Services are the components that interface our application with the database.

In the interface, we describe three methods:

• createData: takes a DataInDTO as input and creates a new entry in the database
• getAllData: returns a list of all the data entries in the database
• getDataByLocation: takes as input a String and returns a list of all the data entries in the database that match the location passed (in our case the location is the name of the Particle device or coreId if we use k6)

Next, we can proceed with the implementation of our DataServiceDefault.

As we can see, first we need to create a connection to the database using:

InfluxDBClientFactory.create(connectionUrl, token.toCharArray(), orgId, bucketId)

Then, in createData, we can write data to Influx (asynchronously by default) using:

writeApi.writeMeasurement(WritePrecision.NS, data);

Finally, we provided two methods to retrieve data from InfluxDB. The first one, getAllData, synchronously selects all the data in our bucket. The second one, getDataByLocation, filters data based on the location tag.

Now, we have to update the DataOutDTO class so that its constructor will take a Data Object and not a DataInDTO. This will be the result:

Finally, let’s update the POST endpoint so that it persists data to InfluxDB. Then we also want to add a new GET endpoint that returns a list of all entries and another one that returns all the entries for a given location. This is the final result:

Let’s test our code!

First make sure you correctly configured your Particle device or k6, as described in our previous tutorial. If you’re using a Particle Console webhook, don’t forget to refresh your endpoint with ngrok and change it in the webhook configuration. If everything is set up correctly, in a few seconds you should be able to see data coming by going to localhost:8080/api/v1/data/all.

To conclude, you can open the InfluxDB Cloud console and see your data from there too. You just need to open the Explore section and select your bucket and measurement that you want to see. Then you can click submit and the data will show up in the chart.

There are many parameters that you can configure in the InfluxDB Cloud Data Explorer. We will let you have a look at them and feel free to post a question in you have any other questions about it.

We made it to to the end of this Getting started with Quarkus and InfluxDB to ingest sensor data from a Particle device tutorial! In the second part of the tutorial, we went through a general overview of InfluxDB and their managed cloud service, we set up our InfluxDB instance and bucket, and we modified our Quarkus application to persist data to InfluxDB. We went through a lot of tools and new concepts. Adapting to time series databases might not be that straightforward, but we hope we gave you a head start for your next big project.

Thanks for reading this tutorial! Let us know what you think about it or if you had any issues in the comments below.

If you want to know more about how we use Quarkus, Influx, and a lot of other cool tools at Wepo, don’t forget to visit our website and get in contact with us!