Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

toolchain install fails in the nightly Docker image #2542

Closed
Rantanen opened this issue Oct 30, 2020 · 13 comments
Closed

toolchain install fails in the nightly Docker image #2542

Rantanen opened this issue Oct 30, 2020 · 13 comments
Labels
bug

Comments

@Rantanen
Copy link

Noted in CircleCI builds and reproduced with local docker:

$ docker run rustlang/rust:nightly rustup toolchain install nightly --allow-downgrade -c rustfmt

I was expecting this to go back to 2020-10-24 build, which should have a working rustfmt. It seems like rustup is attempting to do that, but still fails with an error indicating rustfmt is missing.

info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-30, rust version 1.49.0-nightly (6bdae9edd 2020-10-29)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-29-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-29, rust version 1.49.0-nightly (31ee872db 2020-10-28)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-28-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-28, rust version 1.49.0-nightly (07e968b64 2020-10-27)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-27-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-27, rust version 1.49.0-nightly (fd542592f 2020-10-26)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-26-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-26, rust version 1.49.0-nightly (4760b8fb8 2020-10-25)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-25-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-25, rust version 1.49.0-nightly (ffa2e7ae8 2020-10-24)
info: downloading component 'rustfmt'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: removing previous version of component 'cargo'
info: rolling back changes
error: component 'rustfmt' for target 'x86_64-unknown-linux-gnu' is unavailable for download for channel nightly
Sometimes not all components are available in any given nightly.

A similar test outside of Docker worked just fine:

$ rustup toolchain install nightly --allow-downgrade -c rustfmt
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-30, rust version 1.49.0-nightly (6bdae9edd 2020-10-29)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-29-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-29, rust version 1.49.0-nightly (31ee872db 2020-10-28)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-28-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-28, rust version 1.49.0-nightly (07e968b64 2020-10-27)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-27-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-27, rust version 1.49.0-nightly (fd542592f 2020-10-26)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-26-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-26, rust version 1.49.0-nightly (4760b8fb8 2020-10-25)
info: skipping nightly which is missing installed component 'rustfmt-preview'
info: syncing channel updates for 'nightly-2020-10-25-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-25, rust version 1.49.0-nightly (ffa2e7ae8 2020-10-24)
info: downloading component 'rustfmt'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: removing previous version of component 'cargo'
info: removing previous version of component 'clippy'
info: removing previous version of component 'rust-docs'
info: removing previous version of component 'rust-std'
info: removing previous version of component 'rustc'
info: installing component 'rustfmt'
info: Defaulting to 500.0 MiB unpack ram
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
 13.6 MiB /  13.6 MiB (100 %)  11.9 MiB/s in  1s ETA:  0s
info: installing component 'rust-std'
 22.3 MiB /  22.3 MiB (100 %)  11.4 MiB/s in  2s ETA:  0s
info: installing component 'rustc'
 55.1 MiB /  55.1 MiB (100 %)  12.3 MiB/s in  4s ETA:  0s

  nightly-x86_64-unknown-linux-gnu updated - rustc 1.49.0-nightly (ffa2e7ae8 2020-10-24) (from rustc 1.49.0-nightly (6bdae9edd 2020-10-29))

Rustup status in the nightly Docker image:

$ docker run rustlang/rust:nightly rustup --version
rustup 1.22.1 (b01adbbc3 2020-07-08)
$ docker run rustlang/rust:nightly rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /usr/local/rustup

nightly-x86_64-unknown-linux-gnu (default)
rustc 1.49.0-nightly (31ee872db 2020-10-28)
@Rantanen Rantanen added the bug label Oct 30, 2020
@kinnison
Copy link
Contributor

I have encountered odd errors in CI jobs where the underlying docker container has a rust toolchain in which is being upgraded. I'm not sure why the error message would be that odd, but certainly it's likely to not be the underlying error. Try adding --verbose to see if we can narrow down what is actually failing there.

@Rantanen
Copy link
Author

Was trying that earlier, but I forgot that argument is position specific so it doesn't work if appended to the end of the command but needs to be before the subcommand instead! (A slight annoyance of mine concerning clap.)

https://gist.github.com/Rantanen/c226dc744ba8af62e837610a3c50cd9a

Interestingly rustup manages to download rustfmt from the 2020-10-25 URL just fine but still errors out on rustfmt not available.

Just tried with the official rust:1.47 image and the same command succeeded just fine.

The temp file creation seems weird in the nightly version though. This is the relevant lines from the 1.47 log:

info: installing component 'rustfmt'
verbose: creating temp directory: /usr/local/rustup/tmp/prggu1764vwd4zxg_dir
verbose: creating temp file: /usr/local/rustup/tmp/3fmtuooeq7zgnwxg_file
verbose: creating temp file: /usr/local/rustup/tmp/_zy0a5fnbwc87ef1_file
verbose: deleted temp directory: /usr/local/rustup/tmp/prggu1764vwd4zxg_dir
verbose: deleted temp file: /usr/local/rustup/tmp/6qn_ifs9cypck1s8_file
verbose: deleted temp file: /usr/local/rustup/tmp/2397djxp_qjf8bpr_file
verbose: deleted temp file: /usr/local/rustup/tmp/1yja72v29jw8dje__file
verbose: deleted temp file: /usr/local/rustup/tmp/sagh5gco8fd5ezk4_file
verbose: deleted temp file: /usr/local/rustup/tmp/3fmtuooeq7zgnwxg_file
verbose: deleted temp file: /usr/local/rustup/tmp/_zy0a5fnbwc87ef1_file
verbose: toolchain 'nightly-x86_64-unknown-linux-gnu' installed

And now that I noted the whole info: installing component 'rustfmt' doesn't appear at all. I ended up trying this without the -c argument and the installation still fails. The end of the log is very similar, but after the pile of creating temp file it ends with:

...
verbose: creating temp directory: /usr/local/rustup/tmp/v5_qjvpxu4rrykvn_dir
verbose: deleted temp directory: /usr/local/rustup/tmp/v5_qjvpxu4rrykvn_dir
info: rolling back changes
error: could not rename component file from '/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/zsh/site-functions' to '/usr/local/rustup/tmp/v5_qjvpxu4rrykvn_dir/bk'
error: caused by: other os error

Might be a duplicate of #2472

@kinnison
Copy link
Contributor

Does rustup have write access to /usr/local/rustup in that docker? You could try shelling in to such a docker and deleting that file too, to see if there's write access to the directory. It's renaming which suggests it's unlikely to be #2472 though I wouldn't refuse the possibility yet.

@Rantanen
Copy link
Author

Rantanen commented Oct 31, 2020 •
edited
Loading

Doesn't seem to be a permission issue. Docker runs rustup as root and looking at the directories, the permissions seem to be in order (root:root, rwxrwxrwx) - umask is 0022 by default so new bits are "only" rwxr-xr-x, but even with umask of 0000 the issue persists.

This seems to be a problem with the rename call. Running rustup through strace yields the following failure (indent mine):

mkdir("/usr/local/rustup/tmp/0atmwj5o89hsjjx1_dir", 0777) = 0
statx(
    AT_FDCWD,
    "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/zsh/site-functions",
    AT_STATX_SYNC_AS_STAT,
    STATX_ALL,
    {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFDIR|0777, stx_size=4096, ...}) = 0
rename(
    "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/zsh/site-functions",
    "/usr/local/rustup/tmp/0atmwj5o89hsjjx1_dir/bk") = -1 EXDEV (Invalid cross-device link)

Trying to do a similar move with mv also yields the same failure, but then mv goes and falls back on write+delete instead of rename:

renameat2(
    AT_FDCWD,
    "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/zsh/site-functions",
    AT_FDCWD,
    "/usr/local/rustup/tmp/0atmwj5o89hsjjx1_dir/bk", RENAME_NOREPLACE) = -1 EXDEV (Invalid cross-device link)
...
openat(
    AT_FDCWD,
    "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/zsh/site-functions/_cargo",
    O_RDONLY|O_NOFOLLOW) = 3
...
openat(
    AT_FDCWD,
    "/usr/local/rustup/tmp/0atmwj5o89hsjjx1_dir/bk/_cargo",
    O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
....
read(
    3,
    "#compdef cargo\n\nautoload -U rege"...,
    131072) = 20676
write(
    4,
    "#compdef cargo\n\nautoload -U rege"...,
    20676) = 20676

The invalid cross-device link issue gets few hits on google - the first ones at least are directly related to Docker. Turns out my results were full of docker issues, because my search query was "invalid cross-device link docker"!

So I'm guessing this is a Rust/Docker issue now, not a rustup one.

@Rantanen
Copy link
Author

https://doc.rust-lang.org/std/fs/fn.rename.html

This will not work if the new name is on a different mount point.

This must somehow conflict with files on the Docker image (ie. on the overlay?) and renaming them into new directories (different overlay?).

@Rantanen Rantanen changed the title toolchain install with --allow-downgrade fails to missing components in the nightly Docker image toolchain install fails in the nightly Docker image Oct 31, 2020
@kinnison
Copy link
Contributor

I believe it's a known issue that docker doesn't support this kind of thing, so it's not entirely unexpected. I'm not sure if there's a way to configure docker such that this doesn't happen, but I've seen this plenty of times when people put preinstalled rustup dockers into CI situations.

@Rantanen
Copy link
Author

Rantanen commented Nov 1, 2020

Yeah, at this point I'm not sure what to do with this. The underlying root cause is an exotic configuration where /usr/local/rustup/toolchains and new directories under /usr/local/rustup/tmp are on different mount points. While that configuration is possible without Docker, I doubt that would ever happen in production without Docker.

So while "rustup fails to install/update/etc. if rustup/toolchains and rustup/tmp are on different mountpoints" would technically be a rustup issue, I'd guess it's rare enough to just declare it as "unsupported" and leave it.

I'll leave that decision to you though. Feel free to close this or leave it open depending on how you feel about the issue.

@rbtcollins
Copy link
Contributor

The problem is that you have two different filesystems in ~/.rustup: this isn't supported. See #2232

If you uninstall rustup entirely and reinstall it in one command in docker it will work fine; our current transactional system depends on single filesystem semantics and won't work with dockers default stacked filesystem behaviour; you can either tune docker, so that rename() works, or uninstall and reinstall rustup entirely.

See also #2417 for a related discussion

@Rantanen
Copy link
Author

Rantanen commented Nov 2, 2020

Yes, this is an absolute duplicate of #2232, sorry. Tried searching for a similar issue, but I guess Github's issue search failed me. Closing.

@Rantanen Rantanen closed this as completed Nov 2, 2020
@rbtcollins
Copy link
Contributor

no probs - sorry that this doesn't work easily, and we will get a fix at some point.

@mckaymatt
Copy link
Contributor

@Rantanen @kinnison You may be able to fix this with RUSTUP_PERMIT_COPY_RENAME from https://github.com/rust-lang/rustup/blob/master/doc/src/environment-variables.md#environment-variables if your problem stems from a cross-link device error on overlayfs. See #2410

@rbtcollins
Copy link
Contributor

Is that in a release yet, I don't think it is.

@mckaymatt
Copy link
Contributor

No, you're right it hasn't been released yet.

This was referenced Sep 13, 2021
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Rantanen @rbtcollins @kinnison @mckaymatt