That 2am conversation on test environment provisioning and connectivity …
Yesterday was my 20th Wedding anniversary. I’m married to a wonderful (and tolerant) lady who’s an artist and far, far removed from the world of IT.
But at 2am this morning we found ourselves both awake. I turned to her, and she turned to me. What else could we possibly talk about at such an hour but test environment provisioning and connectivity?
Okay … let’s rewind 30 minutes and it’ll all make sense …
You see, at 1:30am like thousands of Kiwis, we were fast asleep. Then at 1:33am there was a shrill whine from my wife’s phone. Far louder than even the fire alarm we have in our house.
When we unlocked the phone, she had this message …
It looked like Civil Defence NZ were running a test alert through the system. A bit of weird time to be running it for sure … still we’ve really had major earthquakes at inconvenient times before.\
At 1:34am, another loud shrill as a new message (exactly the same) arrived. What was going on?
We decided to get up, do a toilet pit stop, and check the news feeds. Using Twitter, I reached out to see if anyone else had been pinged — my wife’s phone had gone off, but mine hadn’t. [But then again she’s a Civil Defence volunteer].
It soon appeared we were not alone, although there was no real emergency at hand in New Zealand. At 1:49am came another emergency alert …
At this point, I knew this was not an intentional public test in production. When you do such a test, you tend to do it at a convenient time, and you do it once, and only once.
Talking to my wife as we tried to work out what was happening. You don’t do large numbers of test alerts to the public for a very simple reason, it’s called alarm fatigue, or as she called it “the boy who cried wolf”. It’s like the fire alarm that goes off for no obvious reason once … twice … and on the third attempt you just take out the battery.
When alarms go off for no reason, it erodes trust in them. Indeed as you’ll hear me mention in a future episode of Screen Testing, there was a false alarm in 1983 where Russia almost believed it was under nuclear attack from America, and could have chosen to retaliate. Fortunately for the world, that alarm had a dodgy track record, and so the operator who received it decided to ignore it as “it didn’t make sense”.
Civil Defence New Zealand are no fools — the more test messages they run (especially at this hour in the morning), the more they would erode faith in their emergency messaging system, with people asking “how do I unsubscribe from this?”.
Discussing this with my wife, I realised there was only one explanation. Somewhere someone was running a series of test scenarios in some kind of pre-production environment, completely oblivious to the fact they were waking a good portion of New Zealand … repeatedly.
Given the hour of the alerts, it was obvious this person’s office hours were seriously out of alignment with ours — somewhere in Europe or the east coast of America seemed potential candidates.
When you’re in a pre-production environment, you want things as realistic as possible. That invariably means working with “production-like” data, which can often include a migration of production data with some sanitation/scrambling to protect any sensitive customer info.
But the problem with a pre-production environment comes from it’s very strength — it’s almost an exact duplicate of “the real thing”. That means any communications that come out from it look authentic.
A few years ago I created several accounts using my work email address for a prize draw facility. I can tell when testing is being done in that area because I’ll get the occasional email on a Monday morning saying “you’ve won a prize”. For just a moment, I’ll assume I’m one of those really lucky people who wins the lottery without buying a ticket, then I’ll remember it’s the test environment, and that prize is worth less than Monopoly money!
So the golden rule is to sever or at least filter connections between your test environment and the outside world.
For both our email and our text alerts for our test environments we use trap servers to read the traffic, but absolutely don’t allow anything through (although we do tend to let through any email that uses our company email format). It protects you from annoying and pestering a whole host of your users …
We finally got back to sleep, and in the morning I’d become a minor celebrity at work, as some news features had carried my Tweets. It all was pretty much as I’d expected, Civil Defence New Zealand had come forward to explain that the service provider had been running some tests, which were not intended to come through to actual New Zealanders. In future such tests (if required) will be run at more sociable hours.
Indeed as predicted, the upshot of this was a lot of people getting really angry at Civil Defence and asking how to turn off the feature. Something that was the complete opposite of what Civil Defence New Zealand wanted or intended. Such are the high emotions of users when testing harasses or causes nuisance and misdirection to them.
And if you’re reading this, and were the tester involved — come along and say “hi” when I’m at Agile Test Days in Germany in November. We can have a chat and a photo op! [Do I have an April’s Fools idea for you …]
