Light & Wonder Infrastructure Stack design and deployment
By Ravi Velliyagounder, Director Technical Operations Engineering / iGaming Technical Operations
Light & Wonder Technical Operations (TechOps) build and maintain platforms around the world which deliver aggregated content (digital games) to multiple operators in each jurisdiction. Regulations and licensing requirements dictate the location of the tech stack, and in the vast majority of cases, the requirements for physical infrastructure within that jurisdiction/location. This blog post describes how Light & Wonder architect and deploy this infrastructure.
Infrastructure Platform Selection Principles
For maximum flexibility, our software stack is deployed on a VMWare based private cloud built from common off the shelf (COTS) hypervisors, switches and storage arrays. This provides scalable options, avoids vendor lock in and simplifies the ongoing maintenance, for example, patching, upgrades and capacity expansions. 10G internal networking is standard throughout the stack, with InfiniBand SAN for storage.
Design Principles
The tech stack provides services 24x7 with about 230 VMs across production and stage. Light & Wonder do not own and operate data centres; we partner with co-location data centre providers in various licensed jurisdictions around the world. By design and taking the requirements for each Service/VM from Development, CPU is overprovisioned to 350%, memory is never overprovisioned, and the hypervisor quantity ensures a minimum of n+1 redundancy without the need to shut down any stage services, in the event of either a hardware failure or during patching activity. In practice, actual CPU utilisation is ~15% and memory ~45% within the cluster, providing at least n+2 redundancy. Partially automated DRS is used across the stacks, for all VMs other than those hosting databases.
For storage, we have standardised on solid state drives for all primary storage, with either spinning disk or cloud-based solutions, where regulations allow, for backup and archiving. Thin provisioning at the array level is used for all backing stores, allowing storage to be managed within the array aggregates rather than at the LUN level or datastore level.
Infrastructure Deployment model
The build of any new tech stack is documented in a playbook split in to 5 phases, each phase has a Jira Epic to track progress:
- Phase 1 — Generic infra setup
- Phase 2 — Service specific infra setup
- Phase 3 — Application deployment
- Phase 4 — Pre-production setup and pre go-live checks
- Phase 5 — Post go-live checks
Ansible [to read more about Ansible click here ] is used to deploy and configure all the infrastructure and applications, allowing for automated repeatable solutions across our operational Jurisdictions, and reducing the time to deploy from weeks to days. Bit bucket is used as a centralised repository, with all ansible being under formal source code management and subject to peer review for any merge request.
Physical infrastructure remains a core part of our network at Light & Wonder, and as we expand into new regulated markets, consistency of design and deployment, through automation, will continue to be a key success factor.
The opinions expressed in this blog post are strictly those of the author in their personal capacity. They do not purport to reflect the opinions or views of Light & Wonder or of its employees.
WE’RE HIRING
Find out more about life at Light & Wonder and the roles we are looking to fill: https://igaming.lnw.com/careers/
