Enhancing Automation Infrastructure with DevOps Skills

In the dynamic world of software development, the ability to scale and optimize processes efficiently is crucial. At Glassbox, our journey toward improving our automation infrastructure led us to harness the power of DevOps skills, Groovy, DSL scripts, and AWS EC2 Jenkins workers. This combination has enabled us to create a scalable and robust system capable of executing over 1000 jobs a day, ensuring high performance and reliability.

The Challenge

Managing a high volume of automation jobs - including UI, Mobile, API and performance presented significant challenges. Some of the jobs are executed as part of our CICD, as the product gre- the amount of jobs and jobs complexity grew with it.

Our existing static infrastructure, combined of 3 slave workers in Jenkins, each could carry up to 5 jobs at the same time, was not scalable enough to handle the increasing workload, resulting in performance bottlenecks and inefficiencies. We needed a solution that could dynamically scale, automate processes, and maintain high reliability.

Leveraging DevOps Skills

Adopting DevOps principles was the first step in transforming our automation infrastructure. Key DevOps practices that played a critical role included:

1. Infrastructure as Code (IaC) - By treating infrastructure configuration as code, we automated the provisioning and management of resources. This approach allowed us to create repeatable and consistent environments, reducing manual errors and enhancing reliability. For esablishing this we moved all of our jobs to be executed via groovy, lowering the amount of maintance we needed per job. To create all of those jobs we used DSL script that generates jobs dynamically.
2. CI/CD Integration - Those changes allowed us to integrate the automation into out CICD pipelines and to increase our quality gates and to shift left to detect issues earlier.
3. Comprehensive Monitoring - Implementing robust monitoring and logging systems provided real-time insights into system performance, enabling proactive issue resolution and optimization. We created complex dashboard in Grafana that showed us at every given time our Jenkins performance, health and resources.

4. Alerting - adding slack notifications for issues reduced detection of issues late and optimized our process and debugging.

Groovy and DSL Scripts: Streamlining Automation

Groovy and Jenkins DSL (Domain-Specific Language) scripts were instrumental in automating and scaling our Jenkins jobs.

By Using Jenkins DSL scripts, we defined job configurations programmatically. This approach allowed us to manage job definitions in a version-controlled manner, making it easy to replicate and modify jobs across different environments. Since we had many diffrent needs for jobs, this saved a lot time and created a great convention to follow.

Groovy scripts were used to run out automation tests within Jenkins, all while using our version-control to ensure things are operating smoothly. Groovy is our main pipeline tools, from running complicated jobs that perofrm many tasks, such as pulling from repositry, pull docker image from hub, run tests and perform elaborate post actions (publish reports, clean workers, send emails and alerts) to configuring Jenkins jobs to managing plugins and performing routine maintenance tasks, Groovy scripts provided the flexibility and control needed to optimize our automation processes.

AWS EC2 Jenkins Workers: Achieving Scalability

To address the need for scalability, we utilized AWS EC2 instances as Jenkins workers (slaves). This approach offered several advantages:

1. Dynamic Provisioning by leveraging AWS EC2, we could dynamically provision Jenkins workers based on the current workload. This ensured that we had the necessary resources to handle peak loads without over-provisioning during quieter periods.
2. Cost reductions, the ability to scale resources up and down based on demand resulted in significant cost savings. We only paid for the resources we used, optimizing our expenditure.
3. A very important one for us was high availability, AWS EC2 provided reliable and a highly available infrastructure, ensuring that our Jenkins workers were always operational and capable of handling the job queue efficiently.

You can read more about this plugin here.

The Results

The integration of DevOps skills, Groovy and DSL scripts, and AWS EC2 Jenkins workers has led to remarkable improvements in our automation infrastructure:

• Enhanced Efficiency: Automation of job configuration and management reduced manual intervention, streamlining our workflows.
• Scalability: Dynamic provisioning of Jenkins workers enabled us to handle over 100 jobs a day seamlessly, adapting to workload variations.
• Reliability: Robust monitoring and logging ensured high system reliability and quick issue resolution.
• Cost Savings: Optimized resource utilization translated to significant cost savings, as we avoided over-provisioning and paid only for the resources we used.
• Alerts and Monitoring: Enabled us to be informed on issues when they happen and quickly resolve them.

Conclusion

By leveraging advanced DevOps practices, Groovy, DSL scripts, Grafana and AWS EC2 Jenkins workers, we transformed our automation infrastructure into a scalable, efficient, and reliable system. This transformation not only meets our current needs but also positions us to tackle future challenges with confidence, ensuring continued excellence in our software development processes.