Date — Time
“Time is a drug. Too much of it kills you.” (Terry Pratchett, Small Gods)
Dart 2.0 will be a breaking release, and we will use the opportunity to clean up some of our core libraries. We can’t do major refactorings (not that we wanted to), but small improvements and cleanups are possible. One of the affected class is DateTime (https://api.dartlang.org/stable/1.24.2/dart-core/DateTime-class.html).
In this post I’m taking the opportunity to highlight all the difficulties that even simple implementations of date/time classes have to deal with. It’s important to note that Dart doesn’t intend to provide a complete calendar library with its core libraries. A complete calendar library is complex and not needed for most users. As such it should live in a separate package.
Dart DateTime — EcmaScript Date
Dart’s DateTime class was inspired by EcmaScript’s Date class. However, despite many similarities, there are some significant differences between these two classes. For users familiar with the EcmaScript version, this section contains a small summary of them. This should make it easier to follow the remainder of this document.
For both languages a date-time is just a point in time. Specifically, it measures the time since Unix’ epoch (1970–01–01 00:00:00 UTC). In Dart we use microseconds, whereas EcmaScript uses a resolution of milliseconds.
For every DateTime instance, Dart furthermore stores a bit that keeps track of whether the user wants to treat that object as a “local” or “UTC” instance. The local version takes timezone and Daylight-saving time into account, whereas UTC never has any offset. An instance in Dart is either in local time, or in UTC time. The EcmaScript Date doesn’t make that distinction and provides two versions of all methods: getDay() and getUTCDay(), getMonth() and getUTCMonth(), toString() and toUTCString(), …
Dart’s DateTime class is immutable. There are no setX methods on it. On the one hand this is cleaner, but sometimes the setX methods are convenient. In Dart 2.0, we intend to improve the DateTime class by adding a with method that will provide a modified copy of the receiver instance.
Finally, in Dart, months are 1-based. That is, a new DateTime(2017, 1, 20) returns a date in January and not in February (as would be the case for new Date(2017, 1, 20) in EcmaScript).
Different Offsets
Not everyone seems to agree what the best offset for DST should be.
Lord Howe Island only changes by 30 minutes. On the other end of the spectrum Troll Station (in Antarctica) changes by full 120 minutes. Initially this sounds plausible: near the south pole the daylight varies immensely and there is really much more light in the summer than in the winter. However, Troll Station changes their times in the opposite direction: 8am during DST is later than when DST is not observed.
As it turns out, there is a good explanation for it (not that this makes it easier to implement): Troll Station is a Norwegian research station. The station is so far south that, in the winter, there isn’t any light anyway, so they decided to align their watches with Norway for that period to make communication with Norway easier. During Troll Station’s winter, Norway goes through its summer, where it observes DST: CEST (same as Paris). Troll Station thus needs to change their clocks by two hours in the opposite direction of every other country in the southern hemisphere.
Any test that deals with local times needs to take into account that they might run in one of these zones.
Which One?
A common task is to ask the system for a date-time with given values. For example, new DateTime(2017, 10, 20, 15, 24). In UTC this is never a problem. However, in local time, there are many challenges. The first one is to actually do the correct computation. Operating systems tend to do most of the work, but there are limitations. We will have a look at some of these difficulties later.
The second challenge is, that during DST switches there are sometimes two possible date-times with the same values. When the clock is set back an hour (or 30 minutes, or 2 hours), the same date-time values are reused.
The other direction isn’t much easier. Say DST happens at 2:00am, and the clock is set forward to 3:00am. What should happen when the user asks for the date-time at: 2:00, 2:30, and 3:00?
Let’s look at 2:00 and 3:00 first. Remember, that a Dart DateTime only stores a timestamp (represented by the microsecondsSinceEpoch value), and whether the instance is in UTC or local time. This means that the instances for 2:00 and 3:00 must be the same. When decomposing or printing the value, the library has to make a choice: decompose it as 2:00, or as 3:00.
When running this program in the US we get the following output:
TZ=America/Los_Angeles dart dt.dart
2017–03–12 03:00:00.000
2017–03–12 03:00:00.000Unless we change Dart to remember what time value was used to construct the date-time, there is no way to know what output is preferred. As such, both values are treated the same, and the fact that the user initially asked for a date-time with a 2:00 clock is lost.
This is not just a theoretical problem. Some countries, for example Brazil, do their DST switch at midnight. While very few users create date-times with times set to 2:00, users do create date-times at 00:00. This happens most often, when date-times are used for calendar dates. The most convenient time is then 00:00.
For example, a user might want to represent the date when the Berlin wall fell as follows: var berlinWallFell = new DateTime(1989, 11, 9);. With this approach, the constructed date-time falls directly on the DST switch.
In Brazil, both 00:00 and 01:00 are valid during the DST switch, and Dart (as well as many other systems) choose 01:00:
$ TZ=Brazil/East dart brazil.dart
15
1At least the date is still correct. This didn’t use to be the case: older versions of Dart actually move one hour back:
$ TZ=Brazil/East dart-1.24 brazil.dart
14
23This bug wasn’t unique to Dart. Many (if not all) EcmaScript implementations had the same issue. Afaik, Safari still prints the incorrect 14 and 23.
The following lines from V8’s Date implementation shows the reason for this widespread bug:
Unfortunately, this comment isn’t completely true either, since it assumes a 60 minute DST change. It’s good enough for most places though, and a similar fix was recently added to Dart.
Coming back to our original DST switch at 2:00. What should happen when a user asks for an invalid date-time, at 2:30?var invalid = new DateTime(2017, 03, 26, 02, 30).
There is never a point in time when the clock shows 2:30. The most reasonable interpretation is thus: “30 minutes after 2:00”. Since 02:00 and 03:00 are the same, this means the user gets a date-time at 03:30. This leads to the interesting situation where a date-time at 02:30 is after 03:00.
$ TZ=Europe/Paris dart /tmp/invalid.dart
dt1 is after dt2?: trueInvalid date-times also need to be considered when users over/underflow values. When a user asks for a date-time at 02:59:60, then there are two interpretations: “59 minutes and 60 seconds after 2:00”, or simply “03:00”. It turns out that the Linux C library makes the distinction:
In the European timezone this gives:
$ TZ=Europe/Paris ./a.out
The two date-times are not the same.Changing
Another problem date-time implementers face, is that these DST rules are not stable. Countries change their rules surprisingly often, and sometimes at very short notice. Egypt, for example, changed their rules three times in as many years.
In 2009 it observed DST between the 24th of April and the 21st of August.
A year later DST started on the 30th of April, was suspended between August 11 and September 10 (because of Ramadan). Finally, on the 1st of October Egypt then went back to non-DST.
In 2011 Egypt then didn’t observe any DST at all. Since then Egypt stayed with non-DST except for 2014 where it did again the 4 changes in one year.
The 2010 change caused a variety of problems. Windows systems couldn’t get updated in time and didn’t respect the DST rules. In fact the decision was so late that in April 2010 it wasn’t even clear yet when DST would end. From https://www.timeanddate.com/news/time/egypt-starts-dst-2010.html (published on 21-Apr-2010):
The daylight saving end date is not confirmed, according to Egypt’s State Information Service.
Windows wasn’t the only victim. Because of an optimization, Chrome’s JavaScript engine V8 also ran into issues. In 2010 the most popular JavaScript benchmark was SunSpider. One of the benchmarks exercised a lot of Date operations that required DST lookups at different (but relatively close) times.
The V8 developer thus implemented a shortcut: instead of doing expensive system calls for every request it would build up a table of intervals. If, for two given date-times, the DST offset was the same, and the two date-times were close enough together, it would just assume that all date-times within that interval would have the same DST offset.
This makes sense: if a place is following DST on the 1st of May, and still followed it on the 5th of June, then it’s reasonable to assume that all days in between also follow DST. At least, that’s what the developer (now my manager :) thought.
When Egypt changed their DST offset four times in a year, the period between the changes was so short that V8 missed some of them. The fix was easy: decrease the maximum size of an interval in which V8 is allowed to assume that the DST offset doesn’t change:
Note that Egypt also changed their time at midnight which leads to the same problems as in Brazil.
Non DST Changes
DST is clearly the most common source of complications in date-time implementations, but they are not the most difficult ones…
Samoa and Tokelau
In 2011 Samoa and Tokelau didn’t just change by an hour, or two. They skipped an entire day.
Samoa and Tokelau are located near the dateline, which means that some of their neighbors (despite having similar times) are on the same date, and others are a day (now) behind. To strengthen their ties with their trading partners New Zealand and Australia, they decided to switch over to the other side of the dateline. This switch only affected calendars since the time was still the same.
The change was announced in May by Samoa, but Tokelau made their decision only in October. Here is the mail thread on the Olson-database mailinglist, which was clearly notified a bit too late (the discussion happens 14 hours before the change):
Alignments
Around the year 1900 many countries started to align their clocks. The corresponding changes are especially difficult because they even change seconds.
For example, Paris changed their clocks by 9 minutes and 21 seconds on the 11th of March 1911, 5 years before the first implementation of DST. Similar changes happened in many countries in that period. For instance Copenhagen turned its clocks forward by 9 minutes and 40 seconds in 1894.
Timezones
Other interesting facts about timezones:
- Some timezones include a half-hour offset compared to UTC. India, for example, is at UTC +5:30. At least one timezone, Chatham Islands, even uses 45 minutes: UTC +12:45 / +13:45.
- Short names for timezones are not unique. For instance “IST” can stand for “Indian Standard Time” (UTC +5:30), “Irish Standard Time” (UTC +1:00), or “Israel Standard Time” (UTC +2:00).
The following image (from Wikipedia) shows regions (except Antarctica) where local clocks have all been the same since 1970.
Stdlib
The best way to get correct date-times is to use the Olson database, which is the result of a collaborative effort to capture all time changes. Since the database changes frequently, programs usually rely on the operating system to do the updates, and then use system calls to do date-time operations. The C stdlib comes with multiple important functions:
These convert between broken-down time representations (in struct tm) and calendar time (seconds since the Epoch, 1970–01–01 00:00 UTC).
The mktime and timegm functions return -1 when they encounter an error. This is unfortunate, since -1 may also a valid value for date-times close to the epoch… One has to also check the errno global to know whether -1 is an error indication or simply the correct return value.
A bigger problem with these methods is, that the standard doesn’t specify a type for size_t, nor does it give any guarantees on the interval in which the conversions work. This leads to inconsistent experiences on different platforms. On MacOS, for example, the mktime routine works only for negative 32 bits (down to 1902), but works for positive 64 bits.
EcmaScript worked around this by introducing “equivalent years”, which do the computations always in a positive range that is (mostly) guaranteed to work. Dart copied this approach.
Equivalent Years
As mentioned in the previous section, many systems only work for a limited range. As a consequence ECMAScript suggests to map the year of the input to an equivalent year (same leapyear-ness and same starting week day for the year) for which the host environment provides daylight saving time information. The only restriction is that all equivalent years should produce the same result.
The actual range is not defined in the ECMAScript standard, but V8’s implementation maps to 2008..2037.
This has some immediate consequences: for years that are mapped into the safe range, DST might not be accurate. For example, dates might implement DST before daylight-saving time was even invented. On d8 (the command-line version of V8) we have:
d8> new Date(1570, 07, 10)
Mon Aug 10 1570 00:00:00 GMT+0200 (CEST)Note the “CEST” which stands for “Central European Summer Time”.
Another problem comes from the fact that EcmaScript suggests (and for many versions “required”) that implementations work with local dates as if they were shifted UTC dates. This makes sense: UTC is much simpler and allows to rely on reasonable assumptions like that days have always 24 hours.
In order to compute a property, like the weekday, a date-time is first converted to UTC, so that the UTC date-time has the same properties as the localized date-time. This just consists of subtracting the full offset.
Similarly, if JavaScript implementations want to get a moment-in-time from a decomposed localized date, they first compute the UTC date, then ask the system for the zone-offset at this time and finally add the value to the UTC-value time they had computed. (This explanation takes some shortcuts, but as described in an earlier section, the spec shouldn’t be taken literally anyway.)
Together with equivalent years, there is an interesting corner case, though. It can happen that JavaScript engines start with a value in the safe range, then adjust the input to account for the local timezone, which pushes them out of the safe range. (Or the opposite). If the offset is still the same, that doesn’t matter, but there are many countries where the rules have changed.
Let’s look at an example: the 1st of January 1970 in London. In 1970, London actually observed summer time, even during the winter. It looks like Firefox uses negative values as the cut-off values for the equivalent range, which leads to the following output for jsshell (Firefox’s standalone executable for the JS engine):
TZ=Europe/London js
js> new Date(1970, 0, 1, 0, 0, 1).toString()
“Thu Jan 01 1970 01:00:01 GMT+0100 (BST)”Note how we asked for a date-time one second after midnight, but got a Date that is one hour later. Both V8 and Dart have similar issues (just at different times).
For example, for v8:
$ TZ=America/Santo_Domingo v8
V8 version 6.2.0 (candidate)
d8> new Date(1969, 11, 31, 20, 0, 1)
Wed Dec 31 1969 21:00:01 GMT-0300 (-0430)Note that requested time was 20:00:01, but the result prints 21:00:01.
Dart has a much bigger safe area, so the issue is only visible for dates that are much earlier:
$ TZ=Australia/Canberra dart canberra.dart
1901–12–14 06:45:53.000
1901–12–14 07:45:52.999As can be seen, the two dates should be one millisecond apart, but the output differs in more than one hour.
We are actively working on fixing this issue in Dart.
Recommendations
As a general recommendations, use UTC dates whenever possible. With UTC dates one can trust the usual assumptions. In particular, a day is guaranteed to have 24 hours.
In Dart, use the DateTime.utc constructor, or set the isUtc flags to true. This is especially useful when using DateTime as a calendar-date class.
Be careful when using dt.add(new Duration(hours: 24)) to get a date-time with the same clock as the receiver. During DST changes, a day doesn’t have 24 hours, and this computation would not yield the desired result.
Test your programs with different browsers. Safari, Firefox and Chrome all behave differently for edge cases.
Test your programs in different time zones. I recommend to test at least in the following zones:
- Europe/Paris
- Antarctica/Troll
- Egypt
- Australia/Canberra
- Australia/Lord_Howe
- America/Los_Angeles
- Asia/Calcutta
- Brazil/East