how to change XS on a deployed system

[warner]

What is the Problem Being Solved?

How do we deploy a fixed/improved xsnap or XS engine once we've launched the chain? Or more generally, for any deployed swingset kernel (with at least one vat running, i.e. all of them), how do we take advantage of bugfixes, security fixes, performance improvements, or new features, in the XS javascript engine or the xsnap program we use to host vat workers? While maintaining consistent/deterministic behavior among the members of a consensus group?

I've been assuming that the only way to do this safely will be to add a "which version of XS should we use?" field to each vat's stored options, where the current missing value means "the first one", so every vat will keep using the same XS until the vat is upgraded. Then we'd do a binary upgrade to add a new version of XS (giving us two to choose from), then do a baggage-style upgrade of each vat, where the new version uses the same vat bundle but the most recent XS version. That would provide fully-identical vat behavior independent of new XS versions up until the vat upgrade, and we can tolerate any amount of different XS/xsnap behavior because it only appears in the second incarnation (and all validators perform the vat upgrade at the same time).

@arirubinstein and @mhofman pointed out that 1: this is painful, and 2: might not be necessary. Their proposal, for relatively small XS changes, would be:

• all validators do a binary upgrade (i.e. replace their agd with a new version) at a specific block height, as we'd do for any changes to cosmos or the kernel
• when the new agd version starts, all vat workers will be restarted with the new xsnap and XS code
  • they'll reload their most recent heap snapshot, then replay the transcript suffix, just like they would in a normal reboot, but this time with different XS

Of course, this approach only works if the new XS is sufficiently similar to the old one:

• the new XS must be able to load a heap snapshot that was written by the old XS
• the transcript suffix replay must produce the same syscalls and organic GC timing as the old one
  • because of our known syscall sensitivity to virtual/durable collections being GCed, if the new run causes organic GC to happen earlier than the original, it might try to page in a collection Representative during the replay, causing vatstoreGet syscalls that do not exist in the transcript (nor do the results they want back), causing the replay to fail
  • small timing differences might be tolerated, e.g. if the organic GC still happens in the same crank as before, and between two existing syscalls, just slightly earlier or later, the syscall trace would remain the same
  • also, vats which are not very active, or whose deliveries do not do very much, might not experience organic GC before a dispatch.bringOutYourDead forces GC, which basically "resets the clock", after which the timing difference doesn't matter
• the transcript suffix replay must not exceed the per-crank hard metering limit
  • (technically we really mean that the replay must exceed the limit where the original exceeded it, and not exceed it where the original did not exceed it)
  • (but if the original deliveries exceeded the limit, the vat would have been terminated, and thus not around to be replayed)
• metering differences that do not exceed the per-crank limit aren't problem, even if we were metering our vats (which we aren't yet), because we ignore metering during replay

To test this thoroughly before performing an upgrade, we'd need to know the state of each vat at the point of upgrade. And since we schedule upgrades ahead of time (at a particular block height), we could not reliably predict that state, making such upgrades kinda risky.

The risk could be removed by including a kernel-wide dispatch.bringOutYourDead (to every vat) as the last thing done before the old version is shut down. #6263 is about draining the swingset queue entirely at this point, which might take a significant amount of time, but if we're only changing XS, the vats.forEach(vat => vat.BOYD()) would probably be sufficient, and might complete in just a few seconds.

Larger XS Upgrades

If the XS/xsnap change cannot load a heap snapshot created by its predecessor, then we can't just swap out the xsnap binary. Instead, we must perform a vat upgrade, so we have a new incarnation of the vat which uses the second version (it could use different vat code too, or a different liveslots/supervisor/SES: any amount of change is ok as long as it can read the vatstore data of its predecessor, and can fulfill the obligations implied by its exported vrefs).

If we want to avoid having multiple versions of XS at the same time, we'd need the last act of the old agd execution to be a stopVat on all vats. Then the first act of the new agd could be a startVat (with the new engine) on all vats. No heap snapshots would be used across the boundary, and of course all workers are shut down when the application exits.

The downside is that normally a vat upgrade will roll back to the previous version if something went wrong (i.e. if the new version failed to defineKind everything that the predecessor created, or throws an exception during createRootObject or contract setup). If we separate the version-N stopVat from the version-N+1 startVat, there's nothing to roll back to: the heap snapshot is unusable by the new XS image, and the old XS image is gone.

The alternative is the approach I originally sketched out: multiple XS versions, each vat incarnation uses a specific one (which does not change, despite agd or kernel upgrades), and each vat upgrade changes the flags to start using the most recent one. The deployment process would require an agd upgrade first, to add the new version, then a bunch of vat upgrades, to switch to it. That involves the governance committee for each contract, in addition to the chain governance and validator coordination necessary to change agd. It also requires some more creativity in our source control (maybe having multiple packages/xsnap-v1/ / xsnap-v2/ subdirectories in a single repo? eww).

Or perhaps separating out our xsnap into a separate repository(?), so validator operators could do e.g. yarn install -g @agoric/xsnap@$VERSION to make the new version available as xsnap-v2/etc instead of a single version living inside the agoric-sdk source tree. In that case, we wouldn't require an agd upgrade to make xsnap-v2 available, we'd just announce an intention to require v2 at some point. All validators would have to pay attention to the announcement and install it promptly, otherwise when the contract governance vote comes through to upgrade the vat, their node would crash.

[mhofman]

We definitely need to figure out how to handle multiple versions of vat related code, not just xsnap, but supervisor and liveslots, and probably the kernel side adapter that talks to xsnap (vat translators?)

[warner]

(comment removed, apparently I had some sort of cut-and-paste error while editing, and wound up with two versions of the same comment: the one below is the right version)

[warner]

Let's see, there's an interface boundary (at one layer of the abstraction/protocol/technology stack) between the kernel and the worker: e.g. how does it encode syscalls into netstrings, how does the kernel ask it to perform a heap snapshot, that sort of thing. We could imagine changing that protocol (e.g. change from "please write a heap snapshot to filename XYZ" to "please write a heap snapshot on previously-established file descriptor 6" ala #6363), by changing the xsnap binary 's C code (and the kernel-side code that talks to it), without endangering the determinism of the vat's behavior.

There's another boundary (higher up the stack) between the kernel and the liveslots that lives in the worker, which talks about how syscalls are expressed, and marshalling. If it didn't threaten (or we didn't care about) determinism, say in an ag-solo, then we could e.g. switch to "smallcaps" (#6326), or change syscall.send to include "which meter are you drawing from" data, by modifying liveslots and the kernel code, but leaving userspace alone, somehow (probably involving the separate liveslots bundle in #5703).

Given that we do care about determism, there's effectively a boundary between the kernel and the persistent vat image that lives in the worker, the one that includes userspace, liveslots.js, supervisor.js, and everything that gets wrapped into the xsnap heap snapshot. The only way to change the contents of that image is through a vat upgrade (discarding the image and restarting from a new one, only vatstore DB state carrying over). You might do a vat-upgrade because you want different userspace code, or because you want different platform (liveslots/supervisor/lockdown/SES/Endo) code, or both. You might update the platform code every time you do a vat-upgrade, just on principle.

So our initial kernel version K1 will only be able to communicate "vat image protocol" P1, and we'll deploy some set of vats that speak P1. "P1" consists of all the protocol- or compatibility- relevant platform layers: liveslots and supervisor, for sure. The particular version of e.g. SES being used on the worker isn't really part of P1, even though you can't change it without doing a vat-upgrade, because the kernel doesn't care. But the behavior of @endo/marshal very much is part of the protocol, so we'd consider it to be part of the "P1" definition.

Let's imagine one package that contains the kernel (at some particular SemVer version, nicknamed K1), and a separate package "worker" to hold all the code that goes into the worker (so it would import supervisor, lockdown/Endo, liveslots, plus xsnap so it could get the worker binary itself), which starts with a release version we'll call W1. Kernel K1 will have a dependency on worker W1. If the kernel bundles all the worker code when it starts, then it'll have a DB key for W1, and every vat is launches will use W1 (and therefore speaks P1).

Later, we define the "P2" protocol, with some enhancements/changes to P1. We release a worker W2 that speaks P2. We change the kernel package to speak both P1 and P2, and make it depend upon W2 instead of W1, and then we release kernel K2. When an existing deployment is upgraded to K2, it bundles the worker code into a second DB key for W2, but all existing vats keep using W1 from the DB. Any new vats, or any vat upgrades, will start using W2 (and the kernel will speak to them with P2 instead of P1). The kernel retains W1 in the DB until the last vat is upgraded, at which point the refcount drops to zero and it can be deleted (maybe). (We'd have one refcount held on W2 by virtue of it being the default for new vats).

A brand new deployment of K2 will bundle W2 and use it for all vats, and won't have a copy of W1 in its DB.

That kinda implies that the bundled/stored W1 also includes the binary for xsnap, in addition to the JS bundles that need to be installed into it. That's kind of weird: would we store the ELF file in the DB, and write it to a tempdir each time we launch a worker? (obviously we want the OS to use a shared image, to save memory and disk space). Or maybe we use a new swing-store component, much like the existing/old snapStore (until @mhofman 's #6363 work to avoid tempfiles for snapshots lands)?

Another way to think about it is that the supervisor/liveslots/etc bundles are part of the worker/xsnap binary, rather than something the kernel manages. So the kernel says "hey lib-worker version W1, give me the bundles that I'm supposed to send your xsnap", or maybe lib-worker holds and transmits those bundles all by itself.

But in any case, a kernel deployment that has older vat images needs to retain the ability to use the W1 xsnap binary even after the kernel package has been upgraded to use W2 for new/upgraded vats.

We can imagine small changes to W1 that would be handled in different ways:

• say we make some performance improvement to the C code, which doesn't change the behavior of the vat at all
  • this might be safe to deploy without a vat upgrade
  • the semver on W1 might imply this level of compatibilty, like only being a minor or micro version different
  • maybe (maybe) that could match a micro-version increment of "K1", to tell deployers that it is safe (and beneficial) to perform a validator upgrade to this new kernel version, and that deployments will start using a new xsnap, but that vat upgrades are not necessary (or useful)
• or imagine a change which fixes bugs or security faults, and changes metering, but remains compatible with heap snapshots generated by earlier versions
  • this could be deployed to all workers at the same time, ala the "2: might not be necessary" replay-transcript-suffix scheme described at the top of this ticket

But for anything that 1: cannot start from a heap snapshot, 2: causes GC divergence, or 3: is not compatible with the "P1" protocol, a deployed kernel must either upgrade all vats across the K1/K2 boundary, or must keep a copy of the old worker and bundles until all vats have been upgraded to something newer.

Node.js is not excited about a single package having dependencies on multiple versions of the same package: it can handle dependency graphs like (A->B@v1, A->C, A->B@v2), but neither the package.json version-declaration syntax nor the ES5 Modules import syntax have a way to express "A->B@v1, A->B@v2". The only way I can think of to deal with this would be to use entirely different names for the worker package, like @agoric/worker-v1 (always at 1.0, since we can't ever deploy non-trivial changes without putting them in a different package), and then we release @agoric/worker-v2, so K2 can depend on both @agoric/worker-v1 and @agoric/worker-v2 at the same time.

[warner]

#6596 is about defining a stable package to encapsulate the worker behavior, including XS. I think that covers most of this issue, although it's more like "how to maintain a stable XS on a deployed system despite changes to the kernel or other software". Once we have that stable package, and the kernel package pins a specific version of the worker (no ^1.0.0 wildcards), the question of "how to change XS" will have two answers.

The first answer is for small changes, which are capable of starting from heap snapshots left by earlier versions. We'll express these with a worker version like 1.2.0, which means "I can pick up where 1.0.0 or 1.1.0 left off". The behavior of the vat will be a path-dependent function of the deliveries made to it and the version of the worker package active when those deliveries were made. So all validators must switch from 1.0 to 1.1 to 1.2 at the same time, and they must all follow the same heap-snapshot reload schedule. The kernel package will depend upon a specific version of the worker package, and the chain will run a specific version of the kernel package for each block height (it only switches by governance, and a chain software upgrade).

The second answer is for XS changes that cannot accomodate an earlier snapshot (expressed by an entirely new worker package, e.g. @agoric/swingset-worker-xsnap-v2 instead of -v1), and for changes to SES or liveslots. These require a vat upgrade to deploy, so every vat must be independently upgraded (using a baggage-style upgrade, not necessarily changing the vat bundle, e.g. a "null upgrade"). The deployment sequence will be:

• implement the new @agoric/swingset-worker-xsnap-v2 package
  • (we might start it with version 2.0.0, for clarity: every -v1 package has a 1.x.x version, every -v2 package has a 2.x.x version, etc)
  • (we cannot use @agoric/swingset-worker-xsnap-v1 @ 2.0.0 to express this, because the kernel will need to import both -v1 and -v2 at the same time, and our package managers can't do that)
• make a new release of the kernel which depends on both -v1 and -v2
  • internally, this will change the defaultManagerType from xsnap-v1 to xsnap-v2, so any vats created after the kernel upgrade will use -v2
• make a new chain software release which uses the new kernel
• use governance to perform a chain upgrade, deploying the new kernel but not modifying any existing vats, all of which continue to use @agoric/swingset-worker-xsnap-v1
• use a "Big JS Hammer" governance action to make vat-bootstrap tell vat-admin to null-upgrade core vats like zoe, using the same vat bundle and vatParameters, but changing the managerType to xsnap-v2
• use per-contract governance mechanisms to tell zoe to upgrade each contract vat to xsnap-v2
• to upgrade the static vats (vat-admin itself), we have controller.upgradeStaticVat(), and we'll need some extra code in cosmic-swingset to invoke it at the right time

[warner]

We've backed away from versioned worker packages. Instead, for now, our rule is that we aren't allowed to make incompatible changes to XS or xsnap. We can (carefully) make compatible changes, if we can convince ourselves that:

• 1: replaying existing vat transcripts (which were created by the original XS/xsnap) under the new version won't behave drastically differently
• 2: new vats which are born under the new version will behave similarly enough to pre-existing vats to not cause problems

If we believe that we've nailed down the last of the GC syscall sensitivity, then that allows us to make XS changes which affect GC timing and metering (as long as we don't exceed the hard per-delivery computron limit or the memory-allocation limit in either the original or the replay).

To enable us to make more significant changes, we must first implement a scheme to simultaneously run two different versions of xsnap at the same time. We don't need a full @agoric/swingset-worker-xsnap-v2 package, but we will need two different versions of @agoric/xsnap. Then we'll use workerOptions to remember which version is in use, with some implicit "no data means v1" rule.

I'm sufficiently confident that we can implement this later, and that we can upgrade the kernel (and other parts of agoric-sdk) without changing XS or xsnap. So I'm moving this ticket out of the Vaults milestone.

[mhofman]

I've created a new issue to document an alternative to the multi worker approach for incompatible XS worker updates. I believe it makes it possible to only ever have a single version of XS, and avoid traumatic vat upgrades when updating XS (at the cost of potentially longer chain upgrades)