Release 0.5.26 has massively reduced the data range

[gilmoreorless]

My moment-timezone webpack plugin has a failing build when using version 0.5.26. After some investigation it seems the range of years in the compiled data has shrunk considerably from 0.5.25.

Compare the packed data files for 2019a and 2019b. The file size has reduced by around 80%, and a quick switch between the two files shows that the count of abbreviations and offset has significantly changed for many zones.

Picking an example at random:

2019a.json: "America/Atikokan|LMT CST CDT CWT CPT EST|66.s 60 50 50 50 50|01212345|-32B5R.w UFdR.w 1in0 Rnb0 3je0 8x30 iw0|28e2",
2019b.json: "America/Atikokan|CST CDT CWT CPT EST|60 50 50 50 50|0101234|-25TQ0 1in0 Rnb0 3je0 8x30 iw0|28e2",

To see the impact of this, I wrote a quick script to log out the extremes of a particular zone, based on the packed data files:

const moment = require('./moment-timezone-utils');
const versions = ['2018i', '2019a', '2019b'];
const name = 'Australia/Sydney';

function logDetails(version) {
  const data = require(`./data/packed/${version}.json`);
  const rawZone = data.zones.find(zone => zone.startsWith(`${name}|`));
  const { untils } = moment.tz.unpack(rawZone);

  const start = moment.utc(untils[0]);
  const end = moment.utc(untils[untils.length - 2]);

  console.log(`
Version: ${version}
Start:   ${start.toISOString()}
End:     ${end.toISOString()}`);
}

versions.forEach(logDetails);

The result of running this script in the root of the moment-timezone repo is as follows:

Version: 2018i
Start:   1895-01-31T13:55:08.000Z
End:     2499-10-03T16:00:00.000Z

Version: 2019a
Start:   1895-01-31T13:55:08.000Z
End:     2499-10-03T16:00:00.000Z

Version: 2019b
Start:   1916-12-31T14:01:00.000Z
End:     2037-10-03T16:00:00.000Z

I don't think this is intended behaviour, and I suspect it might have something to do with differences in zdump on different machines (as the release commit had to change the zdump usage).

[jsnajdr]

The smaller size of the data files in 2019b is expected, see the "Changes in code" section in the announcement: http://mm.icann.org/pipermail/tz-announce/2019-July/000056.html

There used to be extra data that didn't contain any new information, they were needed only to workaround bugs in various older software.

But there is indeed some information missing in moment-timezone 0.5.26.

A few winter/summer dates formatted in Australia/Sydney for various years, version 0.5.25:

1890-01-01T20:04:52+10:04
1890-07-01T20:04:52+10:04
2000-01-01T21:00:00+11:00
2000-07-01T20:00:00+10:00
2037-01-01T21:00:00+11:00
2037-07-01T20:00:00+10:00
2038-01-01T21:00:00+11:00
2038-07-01T20:00:00+10:00

The same dates formatted with version 0.5.26:

1890-01-01T20:00:00+10:00
1890-07-01T20:00:00+10:00
2000-01-01T21:00:00+11:00
2000-07-01T20:00:00+10:00
2037-01-01T21:00:00+11:00
2037-07-01T20:00:00+10:00
2038-01-01T21:00:00+11:00
2038-07-01T21:00:00+11:00

• the weird 10:04:52 offset that was used in Sydney before February 1895 is missing in the new version
• dates after 2037 don't have DST applied

Relevant content of the canonical tzdata file:

# Zone  NAME            STDOFF  RULES   FORMAT  [UNTIL]
Zone Australia/Sydney   10:04:52 -      LMT     1895 Feb

# Rule  NAME    FROM    TO      TYPE    IN      ON      AT      SAVE    LETTER/S
Rule    AN      2008    max     -       Apr     Sun>=1  2:00s   0       S
Rule    AN      2008    max     -       Oct     Sun>=1  2:00s   1:00    D

[gilmoreorless]

The smaller size of the data files in 2019b is expected, see the "Changes in code" section in the announcement: http://mm.icann.org/pipermail/tz-announce/2019-July/000056.html

As it also says in those notes, that only applies if you manually compile the data files with zic -b slim, which the current build process doesn't. In fact, I notice that the build process just uses zic and zdump binaries provided by the local machine, so the "changes in code" section of 2019b isn't entirely relevant. In most cases this won't matter, but there are times when the data and the binaries can get out of sync (e.g. zic on my macOS 10.14.15 machine doesn't contain the -b option yet). Having a release process dependent on environment-provided binaries opens the path for inconsistent release files, depending on who's build the release at the time.

Regardless, large scale changes to data ranges due to a compilation process should be mentioned in release notes and versioned as a breaking change. All this makes me think the changes are unintentional.

[MartinManev]

Why aren't the maintainers looking into this? I'm really worried about the sudden size drop too.
If I download the the data archives directly from https://www.iana.org/time-zones then I can compare them and see that the newer release 2019b is bigger by about ~5KB in size. That makes sense as the newer release is expected to contain more information than the previous.
And if I compare these two:
https://github.com/moment/moment-timezone/blob/develop/data/packed/2019a.json (904 KB)
https://github.com/moment/moment-timezone/blob/develop/data/packed/2019b.json (176 KB)
There is a whopping 728KB difference!
The unpacked versions are also very different (2.03 MB for 2019b compared to 11.9 MB for 2019a).

I'm going to pass updating this package for now.

Edit: gilmoreorless has explained the actual problem and what is going on internally. I just want to bring more attention to the issue.

[gilmoreorless]

This looks to be the inverse problem of #697, which noted a big increase in size with release 0.5.23.

Quoting @mj1856's response:

With this release, #308 is fixed. That bug was truncating data before certain dates, due to differences in zdump output on various systems. So prior to this release, the "full" versions of the data files weren't actually quite full.

This reinforces my belief that the change in release 0.5.26 is due to differing zdump versions, depending on who's built the release.

[gilmoreorless]

@HNDeploys @ellenaua Sorry to be a pain, but the broken release was over a month ago and there's been no response (or indeed any activity in the repository) since then. (I'm pinging you directly purely because you're listed on the release commit.)

[ellenaua]

Will check it today. Sorry for the delay

[nbryan]

I may have a related issue. In 0.5.25 in my Node REPL:

 > m.tz('2043-09-12T04:00:00.000Z', 'America/New_York').format()
'2043-09-12T00:00:00-04:00'

In 0.5.26:

> m.tz('2043-09-12T04:00:00.000Z', 'America/New_York').format()
'2043-09-11T23:00:00-05:00'

[joshden]

I'm observing the same behavior starting in 0.5.26:

import moment from 'moment-timezone'

function logDateTime(dateTime: string) {
    console.log(moment.tz(dateTime, 'America/Chicago').format())
}

logDateTime('2037-09-08T12:34')
logDateTime('2038-09-08T12:34')

With moment-timezone 0.5.25 (tzdata 2019a)

2037-09-08T12:34:00-05:00
2038-09-08T12:34:00-05:00

With moment-timezone 0.5.26 (tzdata 2019b)

2037-09-08T12:34:00-05:00
2038-09-08T12:34:00-06:00

I originally noticed and reported this for the downstream project js-joda that uses time zone data from moment-timezone.

I would also like to point out that java.time is outputting the time zones correctly for 2038 and beyond.

Java 8u231 updated using TZUpdater with tzdata 2019c:

package com.joshden;

import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.time.zone.ZoneRulesProvider;

public class TimezoneStuff {
	private static ZoneId zoneId = ZoneId.of("America/Chicago");

	private static void logDateTime(String dateTime) {
		LocalDateTime localDate = LocalDateTime.parse(dateTime);
		ZonedDateTime zonedDateTime = ZonedDateTime.of(localDate, TimezoneStuff.zoneId);
		System.out.println(zonedDateTime.toString());
	}
	
	public static void main(String[] args) {
		TimezoneStuff.logDateTime("2037-09-08T12:34");
		TimezoneStuff.logDateTime("2038-09-08T12:34");
		System.out.println(ZoneRulesProvider.getVersions("America/Chicago").keySet());
	}
}

And this is the output:

2037-09-08T12:34-05:00[America/Chicago]
2038-09-08T12:34-05:00[America/Chicago]
[2019c]

[micheleangioni]

Honestly I'm worried about the lack of attention to this... it's causing troubles for several people (see #798)

[nbryan]

Yes, this is preventing me from upgrading to more recent versions since my application deals with dates well into the future. The off by one hour issue causes my tests to fail.

[ichernev]

Hey, sorry guys for breaking up your flow. Apparently we didn't know either, but it was caused by Mac/Linux differences for different maintainers. Check #999

Ideally we should move to rule-based data, so this crap doesn't happen. Can you please share your use case (in #999) about why you need timezoned data past 2047, just curious :)

Closing in favor of #999.

BTW 0.5.37 was released on linux so the range is back up :)