Top tips for running Hack days

How the ITV Common Platform team uses Hack days to collaborate, innovate and develop new ideas.

For three days in December the ITV Common Platform Team ran our Hack days — an opportunity for the team to collaborate on new and innovative ideas that go beyond our day to day work. Here’s the story of how we prepared, how we ran them, and what we achieved.

The ITV Common Platform is the platform that hosts our flagship products like the ITV Hub and BritBox as well as our internal services and systems. It takes the best of open source, cloud native and SaaS tooling, and packages it in highly automated, easy to deploy template architectures and infrastructure-as-code patterns. The team is made up of over 30 internal and outsourced product development teams with 600 developers, engineers and product managers between them.

A team opportunity

We already set aside one day a month for training, or to work on a new idea, but these Hack days went further. We wanted to test if a longer period of time would help us collaborate more and enable us to work on a variety of projects that ultimately make our work in the team and ITV better, easier and more innovative.

Preparing for our Hack days

It was important for us to communicate with our stakeholders, such as the product teams, that the Hack days would be a priority for engineers in those three days. We will aim to do this well in advance for future iterations of the Hack days to ensure as many people as possible can participate.

To make the Hack days most useful, we used Trello to collate our ideas in advance, and ran a pitching session the day before we kicked off. Each person voted on the Trello card they wanted to contribute to, taking into account how much they could realistically commit to in three days. We used Trello because this is the tool we normally use for organising our work but you can use whatever tool you find easiest to source!

We picked the six top voted ideas and intentionally chose to work on cards with more than two votes. This way we didn’t commit to too many projects but also made sure it was always multiple people working on an idea. This was key as we had a number of new starters so we wanted to ensure they got to work with a variety of team members. It was also important to encourage people to work with others who they don’t normally collaborate with for the sake of innovation and experimentation as well as fun!

How we ran the days

Each day we kicked off with stand-up for all teams. Those projects working remotely collaborated in their project specific Slack channels. Others, who had gone to the office, met in person in order to progress their ideas, decide what ‘done’ meant for each idea and ultimately where they wanted to get to by the end of the week. We repeated the same exercise each day to ensure people had enough time, support and everyone was clear on their next steps. It was important for them to flag if their time was being monopolised by day to day work as we wanted to ensure everyone could participate fully.

Always build in enough time for Show and Tell!

After a lot of hard work, the mini-teams presented back to everyone at Show and Tell on day 3 of the Hack days. We couldn’t quite fit this and all the questions others had in one hour so in future we will allocate a longer slot or two Show and Tells at the end of the final day. It all depends on the number of projects / presenters you have and how many questions they receive!

Here’s what we achieved

• Hosting the Cloud Carbon Footprint tool — this is a tool to estimate energy use and carbon emissions from public cloud usage. Both components of the tool were successfully deployed to the cluster as part of the Hack days. — This directly ties to ITV’s sustainability strategy and sustainability goals for becoming carbon neutral by 2030, and is one of the projects we can expand on in 2022.
• Testing a non-routable Virtual Private Cloud (VPC) for Elastic Kubernetes Service (EKS) pod network. — We have limited IP address space for new AWS Accounts or products that require extending. To solve this, we implemented a new non-ITV routable address range: 100.64.0.0/16. This gives us 48,000 new IP Addresses to use in our EKS clusters, which allows us to have 16,000 Pods in each AZ. Since these addresses are not routable across ITV we can use the same range in every product. Access outside of this address space is allowed via a NAT Gateway.
• Using the new AWS Control Tower and Terraform integrations to automate the deployment and build of AWS accounts. — AWS Control Tower makes it easier to set up and manage a secure, multi-account AWS environment. This solution would allow us to deploy custom templates and policies to individual accounts and the wider ITV AWS organisation. — The Terraform module allows us to create and customise new accounts that comply with our security guidelines. This allows us to define pipelines for automated and consistent creation of AWS accounts.
• Created a Kubernetes energy rating service that compared each pod’s requested CPU and memory resources with how much was actually used to find wastage. — This can surface data to developers and incentivise them to improve service efficiency
• Graviton build for a Kubernetes service. It could mean up to 40% improvement in price to performance! (that would be worth $600k/year saving for ITV). It could use up to 60% less energy for the same performance than comparable EC2 (that would be worth nearly 34,000kg of CO2/year — you’d have to plant over 500 trees every year to offset this amount of carbon!)
• Onboarding — Creating a who’s who section on our Common Platform documentation pages so new starters can find more about the people in the team more easily.

And there you have it — our experience of Hackdays! Have you tried them in your teams? Let us know and we can compare notes!

In summary, if running your own Hackdays:

• Prepare well in advance to make the most of the dedicated time
• Get everyone involved to get the best possible results
• Share your successes as well as where you got stuck with the wider organisation
• Organise Hackdays regularly for team morale and to encourage innovation and collaboration

Sound interesting? We’re hiring!

ITV’s Common Platform Team is hiring! If you have skills in AWS, Terraform and Kubernetes and want to join an organisation committed to DevOps principles, check out the roles on LinkedIn or on our website.