Nightly version mismatch

[SiegeLord]

Right now, the Windows nightly version is rustc 0.11.0-pre-nightly (6b1426b 2014-05-25 01:11:19 -0700) while the Linux version is rustc 0.11.0-pre-nightly (33c3edd 2014-05-23 00:31:27 -700). The Windows binary nightlies cannot compile the source nightlies, so there's another version mismatch there (there's no obvious way to tell what version the source nightlies are).

This should not happen, and all nightlies should be in sync. It becomes very difficult to port between platforms when the compiler versions are all different.

[huonw]

This is because the linux nightlies are failing (but not the windows ones).

[alexcrichton]

This was just a hiccup in our automation. Adding more automation to ensure that everything is in sync 100% of the time is probably overkill for now, so I'm going to close this.

[sfackler]

Reopening. This keeps cropping up and we should really just buckle down and fix it. https://botbot.me/mozilla/rust-internals/2014-07-09/?msg=17718208&page=8

[brson]

@alexcrichton Think I might tackle this problem before the full release channel automation. Plan:

• add dist-staging/nightly, dist-staging/beta, dist-staging/stable s3 folders
• build master uploads artifacts to staging folders
• after uploading the artifacts, for each artifact we also upload a ${name}.commit-id file, which contains the commit sha
• add doc-staging/nightly, doc-staging/beta, doc-staging/stable s3 folders
• build master uploads docs to staging folders
• after uploading the docs we upload doc-staging/${branch}/commit-id, which contains the commit sha
• every hour a cron job on the buildmaster runs rust-buildbot/dist-promote.py, which checks for a list of artifacts, and that all the commit id's agree, then moves them to the right place.

[brson]

Actually, after writing all that out, it's a lot of work. May not be right to do it first.

[brson]

cc #17398

[brson]

There's a race condition in this process between the time the script checks for the commit ids and moves everything. Without downloading the entire set there may not be a solution.

[alexcrichton]

I think it's also a little more racy in terms of uploading docs because they're entire directory trees as opposed to one file. Here's how I'd tweak it:

• In each commit-id file, store both the commit SHA and the checksum of the tarball. That way we can download all the commit-id files, make sure the commits all match, and then when we download the tarballs we're sure we're downloading the actual tarball
• In each tarball, include the docs. We should do this for distributions anyway and this way we can ensure that we're always uploading docs for the exact same version of the compiler that's being distributed.
• As part of the dist-promote.py script, unpack the tarball and sync docs to the relevant doc folder on S3.

[alexcrichton]

That would involve not having staging areas for docs on S3 as well, you just couldn't view staging docs.

[brson]

Servo wants this for cargo as well.

[brson]

Per #16456 (comment) we don't really need to do the staging on s3, but can do it locally on the build master. This might be cleaner, certainly leaves less room for network-related errors.

[brson]

Let's do the staging locally on the build master to avoid various complexity. For simplicity I don't want to make changes to how docs are produced.

• add dist-staging/nightly, dist-staging/beta, dist-staging/stable folders to the build master
• add dist-staging/nightly/doc, dist-staging/beta/doc, dist-staging/stable/doc folders to the build master
• build master uploads artifacts from the slave to the staging folders on the master
• after each successful slave build, for each artifact the master also writes a ${name}.commit-id file, which contains the commit sha
• after each successful doc upload doc/commit-id is written containing the commit sha
• after each successful slave build we check for a full set of artifacts based on the commit id's, and if so upload to s3 and delete the contents of the staging folder.

This doesn't include the metadata @alexcrichton suggested previously since there's no extra risk of bad checksums or races.

Not sure if we need to add any locks here or not to protect the consistency of the final upload. It may be that the final step is completely serialized w/r/t any slave steps.

[brson]

cc @wycats

[alexcrichton]

@brson sounds good to me, I probably wouldn't bother with locking initially, it shouldn't be too too hard to add one anyway. We could even have the builders upload to another temporary location and the python configuration itself copies it into the staging location (which I think must be guaranteed to be serialized).

[brson]

The build master doesn't always know the names of the artifacts it is producing, particularly for stable releases, so the suggested scheme doesn't work as is because it doesn't know what commit-id files to look for. We might instead create subdirectories of dist-staging/nightly for each nightly builder, each containing a single commit-id file, then merge all the builder directories together for the final upload.

[brson]

I put a change implementing this into production last night but ran into a major complication: nightly builds kick off at different times depending on availability of slaves, resulting in different revs becoming the nightly. As a result the script didn't recognize the build as finished and never uploaded. This is a pretty ugly problem that I don't immediately know how to solve.

[sfackler]

@brson One solution: have the buildbots checkout the first commit after midnight GMT (or whatever time you want) and build that.

[brson]

I've put this back into production. They should all build the same revision now. Still waiting to see some successful builds.

[brson]

Seems to be working.