Add comprehensive support for remote docker. #785

Create and robustly delete temporary data, which is stored in the user data directory under `${data_dir}/cross-rs/tmp`. Therefore, if program execution is ever halted, deleting the directory will automatically cleanup any remaining data. A termination handler cleans up the temporary data on exit by clearing a stack of temporary files/directories, and we have 2 resource handlers that cleanup the temporary data when the object is dropped. The new installer for the startup hooks is `install_exit_hooks`, which installs both the panic and termination handlers. To get the directory containing all temporary data, use `cross::temp::dir()`. An example of using temporary directories and files is: ```rust { // these are all safe due to single-threaded execution let tmpdir1 = unsafe { temp::TempFile::new() }?; let tmpdir2 = unsafe { temp::TempFile::new() }?; let tmpfile1 = unsafe { temp::TempFile::new() }?; let tmpfile2 = unsafe { temp::TempFile::new() }?; for entry in fs::read_dir(tmpdir1.path()) { ... } for entry in fs::read_dir(tmpdir2.path()) { ... } } // cleanup tmpfile2 -> tmpfile1 -> tmpdir2 -> tmpdir1 ``` Note that only these 2 wrappers are provided, since it guarantees the temporary file and directory stack stays ordered and resources are cleaned up as desired.

This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup. It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use: ```bash cross-util volumes create ``` This will create a persistent data volume specific for your current toolchain, and will be shared by all targets. This volume is specific for the toolchain version and channel, so a new volume will be created for each toolchain. Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before: ```bash CROSS_REMOTE=1 cross build --target arm-unknown-linux-gnueabihf ``` Finally, you can clean up the generated volume using: ```bash cross-util volumes remove ``` A few other utilities are present in `cross-util`: - `volumes list`: list all volumes created by cross. - `volumes remove`: remove all volumes created by cross. - `volumes prune`: prune all volumes unassociated with a container. - `containers list`: list all active containers created by cross. - `containers remove`: remove all active containers created by cross. - `clean`: clean all temporary data (images, containers, volumes, temp data) created by cross. The initial implementation was done by Marc Schreiber, schrieveslaach. A few more environment variables exist to fine-tune performance, as well as handle private dependencies. - `CROSS_REMOTE_COPY_REGISTRY`: copy the cargo registry - `CROSS_REMOTE_COPY_CACHE`: copy cache directories, including the target directory. Both of these generally lead to substantial performance penalties, but can enable the use of private SSH dependencies. In either case, the use of persistent data volumes is highly recommended. Fixes cross-rs#248. Fixes cross-rs#273. Closes cross-rs#449.

Required to patch cross-rs#724 without deleting the entire volume for persistent data volumes. A few changes were required: the entire `/cross` mount prefix must be owned by the user so `/cross/target` can be removed. Next, we use the full path to the mounted target directory, rather than the symlink, since `cargo clean` would just delete the symlink. Finally, we've added `cargo clean` to a list of known subcommands, and it only needs docker if we use a remote host.

Updates the persistent volume using a fingerprint of all files in the project, skipping any cache directories by default. If the file modified date has changed, or the file has been added, copy it to the volume and update it. If the file has been removed, then remove it from the host. To avoid a massive command-line argument, we copy a file containing each changed file on a line to the container, and then remove each file by running a script on the container.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add comprehensive support for remote docker. #785

Add comprehensive support for remote docker. #785

Commits on Jun 22, 2022