Multi-AWS Account Logging System with X-Ray, ElasticSearch and Kibana
A logging system is essential for application/infrastructure to be analyzed, monitored and debugged. AWS has created X-Ray, a tool to collect log data from the execution of AWS services.
A logging system coupled with X-Ray can make it possible to analyze your entire infrastructure and thus determine which services consume the most resources, for example, in addition to error monitoring and debugging.
In this article, we will show you how the TrackIt team set up a logging system to track one of its client’s infrastructure execution environment, store the data in an Elastic Search database, and then create graphs in Kibana that helps users gain additional visibility.
Analyze and debug applications with X-Ray
What is X-Ray?
AWS’ X-Ray service allows developers to analyze the state of execution of services: lambdas, step functions, EC2, and other AWS services.
X-Ray can be used to retrieve valuable information. For the execution of a step function, X-Ray will record:
- Execution duration
- Input and output information (payload / output)
- List of invoked services
- Execution status
- Execution date
- and other data points
This information can provide useful insight when displayed on graphs, using Kibana in this case.
More information on X-Ray can be found here.
Example of Data Collected by X-Ray
Here is an example of X-Ray data obtained by an EC2 execution.
More information and examples can be found here.
ElasticSearch and Kibana
What is ElasticSearch ?
Elasticsearch is a Lucene library based engine to store, search and analyze any type of data.
Instead of storing information as rows of columnar data, Elasticsearch stores complex data structures that have been serialized as JSON documents.
An index can be thought of as an optimized collection of documents and each document is a collection of fields, which are the key-value pairs that contain your data.
Here is an example of a document in ElasticSearch:
ElasticSearch is the perfect tool to store our logs! It will allow us to store a huge amount of logs and access them very quickly.
What is Kibana ?
Kibana is a frontend application providing search and visualization of the data indexed in Elasticsearch by setting up graphical representations. Kibana is a user interface for the monitoring of ElasticSearch.
Here is an example of a dashboard using Kibana:
Kibana will allow us to easily visualize our logs stored in ElasticSearch thanks to different graphs: histograms, line graphs, pie charts and more!
How to install Kibana and ElasticSearch ?
The best way to deploy the ELK stack is to use the AWS ElasticSearch Service.
Amazon Elasticsearch Service is a fully managed service that makes it easy to deploy and run Elasticsearch and Kibana on a large scale and cost-effective basis.
This service deploys the ELK suite wherever you want. With ElasticSearch Service you will have direct access to:
- ElasticSearch
- Kibana
For this tutorial, we used the latest current version (7.9) of ElasticSearch Service.
Multi AWS Account
Why use a Multi AWS Account for logging?
Centralized logging is often necessary in large enterprises for a number of reasons, ranging from compliance and security to analytics.
Why use multiple environments :
- Because each environment has a specific purpose, and they require different performance, different data, different users.
- Because we want to avoid possible problems of one affecting others.
Here is an example with four AWS accounts:
- Development account
- Production account
- QA account
- Log account
This is the log account that will receive all the logs from the other accounts to analyze them and push them into Elasticsearch.
How do I add an AWS Account in my organization ?
Adding a new account on AWS is very simple:
1/ Go to your AWS Account
2/ Click on your profile and go to “My Organization”
3/ Add Account
4/ You can choose to invite a pre-existing account or to create one directly
Logging System Architecture
These are the steps that occur when a service triggers the logging system:
- An AWS service will trigger a lambda at each execution and will send its execution ARN (Amazon Resource Name)
- This lambda will fetch X-Ray raw data of the specified execution ARN
- Then it will push the X-Ray raw data and the AWS Account ID (where the service was executed) to the cross account SQS. This SQS Queue is configured to be used by any account belonging to your AWS organization.
- On the ElasticSearch account, a lambda will be triggered by the SQS Queue for each message received. This lambda will analyze the raw data of X-Ray and gather only useful data. Then it will push the formatted data into ElasticSearch. In addition, it will push the raw message into an S3 bucket to save it.
Having collected all the data in an S3 bucket, we can now ‘replay’ the data. A user can manually trigger the lambda for replaying the data, so it will push the raw data back into the SQS queue.
Conclusion
This logging architecture enables detailed analysis, thanks to X-Ray, of your different development environments by providing a visual overview in real time thanks to Kibana and ElasticSearch. In addition, logs are also saved in an S3 bucket to make sure you don’t lose them, while offering a replay feature to reinject logs into the pipeline.
About TrackIt
TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Venice, CA.
TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.
TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build and deploy a customized solution tailored to your exact requirements.
In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.
