Proposal: exit_settings builtin and --on-exit CLI option for customizing exit behavior

[nicksieger]

Describe the Feature You Want

We have a number of requests (#2427, #3059, #4294, #5210, #3831) related to being able to control what happens when tilt up or tilt ci exits. A brief summary:

1. Enable tearing down of resources on exit ((optionally) tear down resources on Tilt exit #2427, Auto-down CLI Flag/Setting #5210). Either:

• the equivalent of tilt down, where the Tiltfile is re-executed (so that if config.tilt_subcommand == 'down': statements could run), or
• clean up the resources without re-executing the Tiltfile -- use the manifests computed from the last Tiltfile result to clean up or disable all resources

2. Provide multiple ways of specifying the behavior: CLI option(s), Tiltfile builtins, user config file
3. Provide custom behavior on exit rather than using tilt down and if config.tilt_subcommand == 'down': statements (Ability to clean up Local Resource on tilt down #3831, Need a way to run script/command/function when tilt process is about to die #3059)
4. There are a couple other CLI flags and features related to behavior on exit that have no Tiltfile equivalent that could be exposed: tilt down --delete-namespaces, tilt up/ci --output-snapshot-on-exit

Proposal

This issue proposes to address the above issues and requests with a new Tiltfile builtin exit_settings() and CLI option tilt up/ci --on-exit.

def exit_settings(
  down: bool = False,
  disable: bool = False,
  delete_namespaces: bool = False,
  callback: Function = None,
  snapshot: Union[bool,str] = False,
)
   """Customize behavior of `tilt up` or `tilt ci` when it exits (e.g., due to interrupt or term signal).
   
   Default behavior is to do nothing, leaving any existing k8s or docker-compose resources running.

   Args:
     down: If true, execute the equivalent of `tilt down` at exit. The `Tiltfile` will be re-executed so that any `if config.tilt_subcommand == 'down':` conditional statements can be run. Mutually exclusive with `disable`: If both `down` and `disable` are true, `down` wins.
     disable: If true, disable all resources at exit. The effect is similar to `down` except that the `Tiltfile` is not re-executed.
     delete_namespaces: If true and one of `down` or `disable` is specified, delete any k8s namespaces that were created.
     callback: a Starlark function to be executed at exit. Can be used to run custom logic and `local` commands. If `down` or `disable` are also specified, the callback function will be executed before resources are torn down.
     snapshot: If true, Tilt will output a snapshot file named `tilt-snapshot-XXXXXX.json` at exit. If a string, represents a snapshot filename to save. A literal `%s` can be used in the pathname and will be replaced with a value that ensures the filename will be unique.
   """

CLI option --on-exit

A common flag --on-exit will be added to tilt up and tilt ci that allows specifying down, disable, delete_namespaces, or snapshot options.

      --on-exit strings  Values: down, disable, delete_namespace, snapshot. Customize Tilt behavior on exit. See `exit_settings` Tiltfile builtin.

In addition to the values stated, --on-exit will allow a string of the form snapshot=snapshot-%s.json to customize the snapshot filename. Equivalent to --output-snapshot-on-exit string option.

Current Behavior

Tilt does not allow much in the way of customizing what happens when tilt up or tilt ci exits.

Why Do You Want This?

Address multiple issues and user requests related to on-exit behavior.

[nicks]

For what it's worth, I think there are 4 general product use-cases:

1. When I start and stop my tilt session, I want it to be completely isolated (no shared build state, no shared runtime state. Very common among CI/testing workflows.)
2. When I stop and start my test session, I want it to reconnect an existing workspace (shared build state, shared runtime state. Very common among data science / ML workloads on a shared dev cluster.)
3. When I stop tilt, I want it to stop consuming all runtime resources (shared build state, no shared runtime state. Very common among people who are running everything locally).
4. My build system is caching dependencies incorrectly, so want to try again without a build cache. (no shared build state, preserve runtime state, very commonly needed when debugging, think docker build --pull or bazel clean)

There are also confounding factors.

• You often want to preserve dev databases across runs unless they're explicitly cleared, so that you have a persistent set of data to work with (i.e., this is why people often tell tilt to preserve volumes with ‘tilt.dev/down-policy: keep).
• You often want to preserve namespaces, because they don't contain a lot of state anyway, and Kubernetes has lots of bugs where it can deadlock on namespace deletion (ask me how I know this).

I'm not necessarily saying that exit_settings is the wrong solution for this. But I do think this is a hard product problem. And that encoding the decision in the Tiltfile, or when you run 'tilt up', is not necessarily the right place for that decision.

[norman-zon]

Another use case for me would be to resume FluxCD-managed resources after shutting down tilt.

[KunJakob]

Chiming in in support for this proposal. While @nicks presents good points about this not being the best overarching solution, Tilt already requires a lot of intervention to get to run properly. Adding exit_settings would only provide flexibility for cases where a correct/clean method is not feasible or practical.