Automating Database Tasks Using Ansible

By Ahmet Caliskur, Data Platform Lead

A little of history of challenges

Up until 2 years ago, our Database Platform servicing our Open Gaming System (OGS) had so many issues and incidents due to the fact that we were managing all our databases manually.

We currently have about 400 live PostgreSQL databases and this number keeps growing as our business is rapidly expanding at a fast pace.

As any techie could see, it was almost impossible to manage all these databases manually with our limited number of amazing DBAs 😊. Below are some of the database challenges we faced which got out of hand because of the scale of our systems.

1) Installation of new databases: We did not have any standard process of installing new databases. Whenever there’s a new environment, all installations were being done manually by the magical hands of DBAs and we had to trust that all databases are installed and configured ok and good to go live.

2) OGS releases: As database changes have been done manually for each environment, we always had issues with new product releases which needed to be fixed manually by DBAs. That caused database schemas to be different for each environment and it was causing a lot of frustration for each team involved in new releases.

3) DB Changes: Because database config/schema changes were being done manually in environments whenever we had an incident, we had no track of the changes/fixes we applied so similar incidents kept happening on different environments.

We had to change the way we work and had to adapt our databases to the fast paced, continuously changing environment ASAP, or otherwise our DBA team could have ended up working 24/7 to fix issues!

Ansible

Ansible has been our saviour! We decided to study Ansible and use it to address database challenges as described above.

In brief, Ansible is a software tool that provides powerful automation for IT professionals.

Let’s look at the main use-cases of Ansible to understand how it works and how helpful it can be for database environments.

Provisioning:

Ansible allows us to install our databases from a central point. Whether it is just a single instance or high availability database setup, now we can easily install databases as standard by using our install playbooks and verify everything we need is in place.

A few examples from our playbook;

As you can see above, now we can set up database packages and directories according to our standards and verify whether they’re in place correctly or not.

Configuration

One of the hardest things is to keep all our database configurations the same as whenever they change, these changes usually need to be applied to all other environments too. In the olden days, DBAs had to write lots of scripts and had to connect to a lot of servers to apply them, it certainly wasn’t ideal as they had to spend hours and hours to apply each config change!

Now, with the help of our maintenance and configuration playbooks, we can keep all these tasks as our standard and apply them wherever we like with just one Ansible command.

As you can see above, all crucial database maintenance jobs became a few lines of tasks for us on Ansible. That way we can keep our PostgreSQL backup, vacuum, partitioning scripts and configurations in one place! Amazing!

Upgrade too!

Furthermore, we also managed to automate our PostgreSQL upgrade process! Currently from 9.5 to version 12, we upgrade our databases using Ansible too so no need to get lost over so many manual tasks during PostgreSQL upgrades too!

Conclusion

I just wanted to give a glimpse into the automation of our systems with this blog post. I know when you look at the examples here, you might think that’s easy, but I can assure you that entire process of automating was a long and complex transitioning period!

First, we had to work on a schema alignment project to create a gold standard schema for all our databases then write our Ansible playbooks by converting difficult database problems into automated scripts.

We currently have many Ansible playbooks for our specific database management needs, and we are still working on automating many areas to further automate the pain points of manual DBA tasks.

Thanks to our automate-first approach with Ansible, our DBA team are now able to use their time more efficiently improving the environments.

Thankfully we have said goodbye to the days of daily firefighting due to inconsistencies and manual changes with our systems!

As result of our increased efficiencies and automation, we have reduced the number of incidents caused by database inconsistencies to almost zero!

Databases are no longer a bottleneck for most of our issues and we can install or upgrade our databases in just a couple of hours. I am incredibly happy and proud of what we achieve at Light & Wonder each day as a company and as a team together! 😊 😎

The opinions expressed in this blog post are strictly those of the author in their personal capacity. They do not purport to reflect the opinions or views of Light & Wonder or of its employees.

WE’RE HIRING

Find out more about life at Light & Wonder and the roles we are looking to fill: https://igaming.lnw.com/careers/