How to Change PagerDuty Without Breaking Your Rota

One of our common complaints is folks adding or removing someone from the PagerDuty rota in terraform, only to find it’s shot all to hell. So how do you avoid that?

It’s all a matter of rotation_virtual_start, available in terraform under pagerduty_schedule and known in the UI as ‘Changes should take effect at’

Pagerduty cares about three things when it looks at these schedules: Creation date, when the schedule was first created; start, which is when the latest schedule was pushed, and really badly named; and rotation_virtual_start, which is when you want the schedule you’ve defined to start. This last one is important.

So, here’s how it works:

• The order in which you supply the users will be the order they rotate in.
• It will always start from rotation_virtual_start, ignoring everything else.
• If rotation_virtual_start is in the future, it will work backwards, up to the current date, and fill in your schedule using this.
• Whatever you do, it won’t change the past. That’s history and you can’t change history (sorry McFly!).
• But! Pagerduty will start calculating from the past if that’s where rotation_virtual_start is set.

Because calculation can start way-back-whenever if you change the schedule, but not the virtual start time, how your schedule looks today might now look different. The past will look the same because it’s already happened, but from now into the future Pagerduty will gladly change everything!

So, once you know all that, you know that the best way to define your new policy is to:

• Ignore start because it’s going to be overwritten anyway.
• Ignore the past because you can’t change it.
• Set rotation_virtual_start to the start of your current on-call period.
• Set the order that you want people to rotate in starting with whoever’s on call now.

Then your rotation schedule will be pushed out from that date, in the order defined in terraform, and everything should stay intact.

Two important notes:

First, If you’ve moved or removed the current on-call person then their replacement will start from now, only. This will look a bit car-crashy going into the past, because nothing there will change even though you’ve re-run the rota.

Secondly, overrides will stay in place, regardless of the moves around them. If you’ve re-ordered the on-call order, overrides might need to be moved too.

DAZN SRE are hiring!

DAZN’s SRE team is expanding, and we’re looking for new folks to join us in the UK and the Netherlands. We’d love to speak to you if this sounds like something you’d like to do!