EchoStars: HTTP Resource Monitor made with Golang

In today’s market, the importance of Go is continually increasing due to its efficiency, ease of use, and capability to handle modern and complex systems. Its growing adoption in leading companies and its role in critical applications make it a valuable tool for software development at scale.

At Hexacta, we are aware of the growth of Go and its technologies, and the increasing significance it holds in the market. Therefore, from the Hexacta Architecture Team (Hat), we decided to form a study group to engage with the GoLang language, gain experience with its syntax, library, and ecosystem, and apply this knowledge in the development of a project.

This project, called “echoStars”, allows subscribing to server URLs and performing pooling at certain intervals, to monitor their status. echoStars enables the monitoring of server status by sending a notification when a server changes state.

What does “echoStars” do?

The main goal of “echoStars” is to provide a simple yet effective solution for monitoring the availability of web resources. Clients can subscribe to URLs they wish to monitor, and “echoStars” takes care of regularly checking whether these resources are accessible and functioning correctly. This tool can be useful in development environments and, with some improvements, in production environments to validate the availability of web resources via HTTP.

The main objectives are:

• A clear interface, as it needs to interact with different systems, it is necessary to have an interface based on standards like REST HTTP and documented with Swagger.
• Simplicity: This project stands out for its simplicity, presenting a layered architecture with data access to a Bold DB whose storage is key-value, stored in a simple file on the operating system’s disk.
• Allowing different forms of notification, currently, echoStart notifies a server state change via email. However, it was designed to work with different types of notifications.

The project has documented the APIs in Swagger, allowing for easy communication with clients who wish to perform operations such as subscribing to a server to begin respective monitoring.

In the future, it could be utilized by developers and companies that rely on the availability of their services or for monitoring a microservices ecosystem.

Technologies and Tools Used

We built this project using the Go (Golang) programming language, which is known for its efficiency and performance, especially in handling concurrency.

Among the dependencies we used, we have:

echo

It is a powerful web framework for building high-performance applications in Go (https://echo.labstack.com/docs). Echo includes a routing module that allows us to define the respective REST methods.

Swaggo

It enables the generation of automatic documentation in Swagger.

Gommon

In our case, we use it for writing in the log.

Gopsutil

Based on psutil (https://github.com/giampaolo/psutil), it is used to monitor system metrics such as CPU, disk space, and memory information, among many others.

Testify

It allows us to implement robust unit tests. In our case, we use it to evaluate the asserts in the tests, and it works similarly to the asserts found in Java’s JUnit.

Bbolt

Bbolt is an excellent choice for Go applications that need a high-performance embedded database and value simplicity and speed in their read and write operations.

Gomail

A library that provides a simple interface for building and sending email messages. We use Gomail in conjunction with Mailtrap (https://mailtrap.io/), a service that simulates the environment for sending and managing emails.

goMock

As its name suggests, it allows us to mock components, and in our case, in the integration tests, we use it to simulate calls to the service layer.

Example:

Similar to many libraries for performing mocks, its syntax is ‘EXPECT’.[Operation]. Return, giving us the ability to create robust mocks and perform testing due to the low coupling of our application.

Project Structure

echoStars is organized into a series of modules that reflect the principles of a layered architecture:

database: This layer handles the connection to the storage component. Here, we configure the properties of this connection by reading from the properties file.

docs: Contains the Swagger documentation. This documentation is generated automatically from the exposed routes using the swag init command at the root of the project.

monitor: The monitor component, upon initialization, retrieves the list of servers available for monitoring. It then makes a request to each server’s URL to evaluate its status, writes to the console based on the server’s state, and updates the database if there is any state change for a server.

notification: A module that will eventually notify the client subscriber. Designed as a service that implements email sending, but can also be adapted to other implementations such as sending text messages.

server: Corresponds to the server components, i.e., information like the URL, subscription status, email to notify, and the URL for subscription validation: ‘UrlHealth’.

routines: A module demonstrating the use of go routines, calling the getSites function with different implementations to compare the functionality and efficiency of using Goroutines. The first two executions of getSites use the synchronous implementation to load HTTP requests in cache and eliminate that variable from comparison. It then executes two calls using go routines, the first using WaitGroups and the second using Channels. The execution time of each of these calls is recorded in the console, demonstrating how using go routines can significantly optimize the code.

sysinfo: A module dedicated to evaluating system metrics.

While this project is open to many improvements, the structure we present allows for the maintainability and scalability of the code, which is very important for any project seeking to follow the principles of clean architecture.

Monitoring Servers with Goroutines

Goroutines are a fundamental feature of Go that makes concurrent programming more accessible and efficient. Their lightness and ability to work in conjunction with channels make Go ideal for developing concurrent and scalable systems. For this reason, we chose this feature to check the status of servers concurrently at regular intervals.

To perform monitoring, the monitor component retrieves the list of available servers, which are passed as arguments to the WatchServers function. This function creates a Goroutine for each server. Each Goroutine makes an HTTP request according to the frequency and configured URL of each server to evaluate its status periodically and record relevant information of each request in the console. If the status of a server changes, it is recorded in the console, and the status change is updated in the database.

Nodejs Test Server

To conduct consistent local tests on different common use cases expected in real environments, echoStars includes an ExpressJs server with several endpoints intended to be consumed by the goroutines.

The available endpoints are:

• http://localhost:3000/healthCheck/success: responds with status 200
• http://localhost:3000/healthCheck/error: responds with status 400
• http://localhost:3000/healthCheck/skecthy: responds with status 400, 10% of the time

Contribution/Conclusion

“echoStars” is a tool that is evolving, so it is open to contributions and improvements. The project is available on GitHub, and any contribution or suggestion is welcome. The URL is https://github.com/emanuelpeg/echoStars

Special thanks to Axel Leanner Salmeron, Juan Pablo Alvis, and Sergio De La Torre for their invaluable contributions to this project. Your dedication and expertise have been instrumental. We deeply appreciate your efforts.