How did we organize our last ManoMano GameDay?
Here at ManoMano, we practice Chaos Engineering for quite some time, so this is not our first rodeo. In this article, you’ll find some insights about the organization of our last GameDay!
Hope it will give you some tips and envy to organize a GameDay for your teammates.
And just in case you missed our previous article on GameDays and Chaos engineering philosophy at ManoMano, here it is !
Defining the topic of this 4th edition
Even with the pandemic behind us, the habits and the world of work are still deeply changed. Many activities have shifted to a remote operation. And during the pandemic, e-commerce activity increased tenfold, as did online activity. Unfortunately, some people have seen this as an malicious opportunity to steal data.
The number of data breaches and the cost associated risen quite a lot since 5 years (fig1).
So we decided to focus on Security for this GameDay. A thief can do some damage by stealing data or use privilege access to launch malicious programs!
We are not the firsts to mix chaos engineering and security… in fact this new approach has been making its debut for a couple of years.
Generating GameDay content
Now that we have the topic and the agreement of the security team to take part, it is time to find material for this day!
Workshops
We planned workshops with security team, then track progress with follow-up meetings. To stimulate the workshops, we were two meeting leaders with the 3 security people. We first find the themes (fig2) and that’s where I am happy to have those guys by my side and not against ManoMano :).
So to list themes permit the security team to diversify a bit the propositions afterward. They manage to find 10 experiments! That’s great, we only needed 3 or 4! I cannot disclose of course our ideas of security threats but I can share to you the elements of the table header:
- Title
- Environment
- Threat goal
- Hypothesis
- Quick modus operandi
- Likelihood
- Severity
- Current level of detection
- Estimated mean time to detect
- Impact on end-user
So we select the most interesting ones and try to offer some diversity. The learning experience offered by the experimentation is an important criteria.
Documentation
Well, now let’s move from the ideas to the actual material, it’s time to get down to the documentation. We give the following template (fig3) to the security to fulfil (I share with you the summary):
As you can see we try to document everything about the experiment and more importantly how to run it. To give some pace about the experiment, we decide to offer some hints during the run of the experiment. And to add to the learning value, we offer also quick story with “Did you know?” (fig4) messages during the experiment.
Tooling
Ok, we have the content, but not how we will run it! To give pace to the event, we choose to play this GameDay as a little race with checkpoints to confirm!
The checkpoints in question are :
- Root cause analysis
- Mitigation
- Post-mortem
So we choose to tweak a little bit a tool dedicated to a well-known contest in security : capture the flag. So we used CtfD to create one flag by checkpoint, and we did it for each of our 3 experiments selected.
Thanks to the tool, we can automate the chaining of the experiments! And we can divide our participants in many teams! This give the sense of competition with one common goal : stop the attackers!
To progress, teams must complete each step (root cause, mitigation, post-mortem). To do this, they must find the rights answers and deliver them to their team slack channel. Then someone from the “Control Tower” deliver a magic token to enter on the tool to access the next step!
This was a manual step, we did not have the time to automate that but we will definitely do it for another GameDay.
Concerning the attacks, I can’t disclose these security threats but all was mostly done via scripting.
Ok the content is here, the way to display content and how to run it too ! So maybe we need to do a little advertising about the event too no ? We still invited nobody at this stage.
Promoting the event internally
Since we have the theme of our GameDay, we can give some ideas to the Studio team at ManoMano to deliver a beautiful Logo.
With this logo, we are set to send nice communications and begin the process of search and order goodies to add some fun to the event!
So when the creation of the goodies were secure, we send a massive communication to all tech people to begin the inscription ! With the help of our Technical Ambassador, the inscription has been sent with typeform, a great tool to follow-up inscriptions and do a nice form.
Setting-up event logistic
To properly run this event, we used a two different runsheets (a run sheet is a list of procedures or events organized in temporal sequence). The first for the day organization and for the second one we zoomed at the afternoon when we planned the experiments to be launched!
Let’s see the first one!
As you can see, we let the morning quiet to welcome people with a breakfast and then we introduce the games with a talk from an external speaker, Dennis Schulte from Steadybit to not name them. :)
The talk was about one of the main goals of the GameDay: Fostering a culture of resilience! You can have a look at this conference during another event here.
And the main challenge for the logistic of this event, was to schedule everything in order to have the correct requirements right on time…
We needed logo & banners from the Studio, goodies before the event. The external speaker soon enough to be able to jot a quick word in the invitation with the summary.
You can find below a global retroplanning to have the big picture of the event roadmap:
Running the event
With the help of my fellow teammates, we got ourselves a complete control tower (4 from Security team and 9 from the Pulse team) as you can see to follow the teams participating to the event:
7 teams with 6 people has been targeted by 3 attacks in the afternoon! So a lot was going on, we tried our best to timebox the attacks.
For this, we wrote another runsheet in the form of slides, (1 slide by step of the attack, root cause / mitigation / post-mortem) with a detailed timeline.
That way you have the course of actions, the communication to make, who is responsible for what and what we want from the participants.
Nothing more to add here, the main objectives were to follow the timing and communicate with each other all the time!
Wrapping up the event
When all attacks were finished, we gathered every teams in a Zoom meeting to recap all the attacks, giving the mic to the attackers 😈, a good way to summarize what we learned that day!
Then we ask a quick feedback from the participants, since the GameDay was cybersecurity themed and contains beautiful challenges, we are not surprised by the response from the participants 😄.
Then the miroboard contains an area to get more structured returns, a quick reminder to grab the goodies since the event was hybrid (remote and on site) and a link to register to do chaos engineering with the tool Steadybit 🛠!
Then we close the event and let everybody rest 🛌 of this tense afternoon!
Key takeaways
I hope this article gave you some tips to run a GameDay, if you have ideas or tools to run during the GameDay, don’t hesitate to share in the comments below!
Here a few takeaways to summarize :
- GameDay experiments should be fun and have a great learning value
- Build an event retro-planning to tackle your tasks effectively
- Timebox everything during the event
- Automate every task you can for the event as, with multiple teams, dealing manually might be a pain
- Nothing is easy for participants, don’t hesitate to give context and hints, a good pace during the game is more entertaining for the participants
- Try to have teams as homogeneous in Tech level as possible (it can be different levels within the team but you must not have an overpowered team)
And to finish this article, some verbatims and highlights of this event 👌
We ❤️ learning and sharing
I took a lot of pleasure to write this article, feel free to post your feedback below and reach out to me on LinkedIn. Whether you had a similar or totally different experience, I’d love to hear about it.
Oh, and by the way: ManoMano is also on Twitter!
