The Worst Of Times
What time is it? There are many ways to answer this question, but is it possible to state an unambiguous absolute time that can be understood by anyone in any context? (Yes, obviously, it’s very easy.)
At Singularity, every day we pull in hundreds of thousands of data points representing minute-by-minute electrical grid activity from sources around the world. Determining exactly when each event occurred is of paramount importance. But rather than adopt a universally supported standard, many of our sources seem more partial to Einstein’s theory of special relativity — it is impossible for two different observers to agree on a concept of absolute time, so why bother trying? Here are the four horsemen of bad timestamps.
No Timezone
The most common problem is the simple lack of a timezone. A timestamp like 2023-01-06 14:57 from an organization responsible for infrastructure that stretches across three timezones is as meaningful as 2023-01-06 mid-afternoon. We are forced to take a guess.
Worse yet, one source provides un-zoned timestamps that turn out to be in EST, and only EST. Despite residing in an area that switches to EDT for half the year, this organization continues reporting in EST straight through the daylight savings period, resulting in a fun groundhog-like bug that only shows its face twice a year.
The Fix: Just label the timezone. (EST) is only an extra 5 characters. If you’re feeling really daring, use an ISO 8601 timestamp with a UTC offset like -05:00.
The Real Fix: Contact the organization responsible and play a fun game of phone tag until they finally break down and tell you the timezone.
Daylight Savings
As you may be aware, daylight savings, or DST, is a conspiracy to prop up the software industry by creating countless hours of unnecessary work. In the UTC timezone there is no daylight savings, and every moment follows the last in a predictable yet monotonous parade. Fortunately for our engineers, most grid operators prefer to report data with timezones that do follow DST, where entire hours may appear or disappear at will. How can this be represented in the data? So glad you asked.
One data provider offers an Excel spreadsheet with a column for each 15-minute interval of each day. When we spring forward in March, there’s a block of blank cells for the skipped hour. Where did it go? Skip ahead to November, and there are four new (DST) timestamps for the repeated hour of the fall: 01:15 (DST), 01:30 (DST), 01:45 (DST), 02:00 (DST).
“No problem,” I hear you thinking, “I’ll just use my handy timezone library to convert this to UTC.”
pytz.timezone("US/Central").localize(parsed_date, is_dst=True).astimezone(utc)NO! WRONG! You’ve fallen for their trap. There is no such thing as 02:00 (DST)— the repeated DST hour is only 01:00-01:59. Here 02:00 (DST) actually represents the end of the 15-minute interval 01:45 (DST)through 01:59 (DST), so you’ll need to subtract 15 minutes before you can convert to a sane time zone. This is not stated in any documentation, but would it be any fun if it was?
The Fix: Use UTC. There is no excuse.
Bad Offsets
This data source just wants to be a little different, a little quirky. Instead of predictable 5-minute-aligned intervals, it’s (almost) always just a few seconds ahead. It feels spiteful — they’ve gone to the trouble of normalizing their data into regular 5-minute segments, but just can’t bring themselves to make it so easy for us.
The Fix: I mean, just don’t do this. Why would you do this?
Chaos
The following is presented without comment:
The Fix: Computers were a mistake, let’s go back to sundials.
