How to use cross with custom toolchains?

[NobodyXu]

Maintainer's Note: The original title was "How to use cargo-bisect-rustc with cross?", however, the issue is more general and deals with working with custom toolchains broadly, or at least specifically custom toolchains that are installed from Rust sources.

---

When running cargo-bisect-rustc -vv --script=./test.sh where test.sh is:

#!/bin/sh

exec cross run --target powerpc64le-unknown-linux-gnu

The cross command failed with:

error: `"rustup" "target" "list" "--toolchain" "bisector-nightly-2022-04-26-x86_64-unknown-linux-gnu"` failed with exit code: Some(1)

If I just run cargo-bisect-rustc -vv without using cross, then everything runs without problem.

Is there any way to fix the problem?

[Emilgardis]

cross needs to know where the toolchain lives, I think it would be possible to support this, unless it's actually already supported.

The error is due to this:

cross/src/rustup.rs

Line 42 in 7599e1a

pub fn available_targets(toolchain: &str, verbose: bool) -> Result<AvailableTargets> {

not working correctly, why it doesn't work I'm unsure.

You should get a better output if you do exec cross run -vv --target powerpc64le-unknown-linux-gnu instead.

also, make sure you have the latest (unreleased) cross, as there have been some changes, but I doubt they would fix this.

My guess is that the way we discover sysroots and targets is not robust enough for bisect-rustc

[NobodyXu]

OK, I've installed the trunk version of cross using:

cargo install --git https://github.com/cross-rs/cross --branch main cross

And cross gives me:

+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu"
Error: 
   0: `"rustup" "target" "list" "--toolchain" "bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu"` failed with exit code: Some(1)

Location:
   src/extensions.rs:26

Edit:

+ rustc --print sysroot
/home/nobodyxu/.rustup/toolchains/bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu

+ rustup toolchain list
stable-x86_64-unknown-linux-gnu (default)
nightly-x86_64-unknown-linux-gnu
bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu (override)

+ rustup target list --toolchain bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu
error: toolchain 'bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu' does not support components: bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu is a custom toolchain

[NobodyXu]

@Emilgardis I think the problem is that cargo-bisect-rustc adds a custom toolchain that cannot add new target.

After reading the cargo-bisect-rustc, it turns out that cargo-bisect-rustc support option --target for cross compiling.

cargo-bisect-rustc -vv --script=./test.sh --target powerpc64le-unknown-linux-gnu

but it gives me the same output:

+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu"
Error: 
   0: `"rustup" "target" "list" "--toolchain" "bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu"` failed with exit code: Some(1)

Location:
   src/extensions.rs:26

[Emilgardis]

for cross to work, you'll need a toolchain that is x86_64-unknown-linux-gnu with your target added. Unsure how it works completely but I don't see why this would not be possible to support, either from an added cfg-flag in cargo-bisect-rustc or with new functionality in cross, e.g a "TOOLCHAIN_PATH" variable

[NobodyXu]

for cross to work, you'll need a toolchain that is x86_64-unknown-linux-gnu with your target added. Unsure how it works completely but I don't see why this would not be possible to support, either from an added cfg-flag in cargo-bisect-rustc or with new functionality in cross, e.g a "TOOLCHAIN_PATH" variable

@Emilgardis The main problem is that cargo-bisect-rustc sets up a custom toolchain, so rustup target list does not work:

+ rustup target list --toolchain bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu
error: toolchain 'bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu' does not support components: bisector-nightly-2022-04-27-x86_64-unknown-linux-gnu is a custom toolchain

So maybe cross can skip checking for available targets for cargo-bisect-rustc, probably by checking whether the toolchain starts with bisector.

[Alexhuszagh]

I'm fairly reticent to add specific exceptions for custom rustc wrappers, however, we might be able to add a command-line flag or determine if the rustc command is not actually rustc? I don't think we should add exceptions for specific crates acting like they're rustc to not be rustc.

Making the logic more general may warrant this functionality.

[Alexhuszagh]

I'm not reproducing this anymore: the error would have to be in rustup::available_targets, which is always run as long as Docker is invoked, which I think is because cargo-bisect-rust actually properly lists the toolchain? I'm not reproducing this anymore. For example, a simple script shows:

exec rustc --version

And cargo-bisect-rustc --script=./script.sh shows that we get rustc 1.63.0-nightly (ca122c7eb 2022-06-13), the default toolchain. Likewise, actually printing out the toolchain versions shows the same thing on the rustc end. This seems to have been patched in cargo-bisect-rustc. Can you check again?

[NobodyXu]

Sure

[NobodyXu]

@Alexhuszagh Still encounter the same error:

nobodyxu@comp:~/tests$ cargo bisect-rustc -vv --target powerpc-unknown-linux-gnu --script=./test.sh
installing nightly-2022-06-15
cargo for x86_64-unknown-linux-gnu: 6.56 MB / 6.56 MB [================================================================================================================================] 100.00 % 1.83 MB/s 
testing...
+ exec cross run -vv --target powerpc64le-unknown-linux-gnu
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu"
Error: 
   0: `"rustup" "target" "list" "--toolchain" "bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu"` failed with exit code: Some(1)

Location:
   src/extensions.rs:26
RESULT: nightly-2022-06-15, ===> Yes
uninstalling nightly-2022-06-15

installing nightly-2022-06-13
rustc for x86_64-unknown-linux-gnu: 1.75 MB / 54.40 MB [===>------------------------------------------------------------------------------------------------------------------------] 3.21 % 885.28 KB/s 1m 
^C

test.sh:

#!/bin/bash

set -euxo pipefail

exec cross run -vv --target powerpc64le-unknown-linux-gnu

[Alexhuszagh]

Can you provide the output of cargo bisect-rustc --version? Mine is cargo-bisect-rustc 0.6.3, which was released on April 20th.

[NobodyXu]

Can you provide the output of cargo bisect-rustc --version? Mine is cargo-bisect-rustc 0.6.3, which was released on April 20th.

Should be the same since I just reinstalled it (because broken libssl.1.1).

I will post these info below.

[NobodyXu]

cross 0.2.1 (7599e1a 2022-04-23)
cargo 1.61.0 (a028ae4 2022-04-29)
cargo-bisect-rustc 0.6.3

[NobodyXu]

rustup 1.24.3 (ce5817a94 2021-05-31)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.61.0 (fe5b13d68 2022-05-18)`

[Alexhuszagh]

I'm reproducing this on stable but not nightly. FYI: I'm using a modified cross binary that's printing the toolchain names during execution, other than that, it's not modified in any way.

Stable

$ rustc --version
rustc 1.61.0 (fe5b13d68 2022-05-18)
$ cargo-bisect-rustc --script=./script.sh -vv
installing nightly-2022-06-15
cargo for x86_64-unknown-linux-gnu: 6.56 MB / 6.56 MB [========================================================================================================================] 100.00 % 9.40 MB/s 
testing...
+ "cargo" "metadata" "--format-version=1" "--filter-platform" "powerpc64le-unknown-linux-gnu"
+ "rustc" "--print" "sysroot"
default_toolchain="bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu"
+ "rustup" "toolchain" "list"
final toolchain="bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu"
+ "rustup" "target" "list" "--toolchain" "bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu"
Error: 
   0: `"rustup" "target" "list" "--toolchain" "bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu"` failed with exit code: exit status: 1

NIghtly

]$ rustc --version                                                                                                                                            
rustc 1.63.0-nightly (ca122c7eb 2022-06-13)                                                                                                                                                         
$ cargo-bisect-rustc --script=./script.sh -vv                                                                                                                
checking nightly-2022-06-14 from the currently installed default nightly toolchain as the last failure                                                                                              
installing nightly-2022-06-14                                                                                                                                                                       
testing...                                                                                                                                                                                          
+ "cargo" "metadata" "--format-version=1" "--filter-platform" "powerpc64le-unknown-linux-gnu"                                                                                                       
+ "rustc" "--print" "sysroot"                                                                                                                                                                       
default_toolchain="nightly-x86_64-unknown-linux-gnu"                                                                                                                                                
+ "rustup" "toolchain" "list"                                                                                                                                                                       
final toolchain="nightly-x86_64-unknown-linux-gnu"                                                                                                                                                  
+ "rustup" "target" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"

I guess the short-term, quick-and-dirty solution would be to use nightly, but there should be a way to fix this on stable.

[Alexhuszagh]

I think the issue is how we get our toolchain name (EDIT: nevermind, I don't think this is the issue), which might make this an issue for non-standard Rust installs too. There probably should be a more robust way to get the toolchain name.

cross/src/lib.rs

Lines 316 to 320 in d90b016

	let mut sysroot = rustc::sysroot(&host, &target, verbose)?;
	let default_toolchain = sysroot
	.file_name()
	.and_then(|file_name| file_name.to_str())
	.ok_or_else(|| eyre::eyre!("couldn't get toolchain name"))?;

This also doesn't seem to work when manually specifying the toolchain:

$ cat run.sh 
#!/bin/bash
exec /home/ahuszagh/git/cross/target/debug/cross +stable run -vv --target powerpc64le-unknown-linux-gnu

$ rustup run stable cargo-bisect-rustc --script=./run.sh -vv                                                                                                 
installing nightly-2022-06-15                                                                                                                                                                       
cargo for x86_64-unknown-linux-gnu: 6.56 MB / 6.56 MB [========================================================================================================================] 100.00 % 9.81 MB/s 
testing...                                                                                                                                                                                          
+ "cargo" "metadata" "--format-version=1" "--filter-platform" "powerpc64le-unknown-linux-gnu"
host=X86_64UnknownLinuxGnu
+ "rustc" "--print" "sysroot"
default_toolchain="bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu"
+ "rustup" "toolchain" "list"
+ "rustup" "toolchain" "add" "stable-nightly-2022-06-15-x86_64-unknown-linux-gnu" "--profile" "minimal"
info: syncing channel updates for 'stable-nightly-2022-06-15-x86_64-unknown-linux-gnu'
info: latest update on 2022-05-19, rust version 1.61.0 (fe5b13d68 2022-05-18)
error: target 'nightly-2022-06-15-x86_64-unknown-linux-gnu' not found in channel.  Perhaps check https://doc.rust-lang.org/nightly/rustc/platform-support.html for available targets
Error: 
   0: couldn't install toolchain `stable-nightly-2022-06-15-x86_64-unknown-linux-gnu`
   1: `"rustup" "toolchain" "add" "stable-nightly-2022-06-15-x86_64-unknown-linux-gnu" "--profile" "minimal"` failed with exit code: exit status: 1

[Alexhuszagh]

Sorry for the many notifications, but I doubt this is an issue in cross itself and seems to be an issue with cargo-bisect-rustc or how it handles toolchains. We can't really handle custom toolchains, since they don't let us add additional targets, etc., and this prevents cross from working with tools that use custom toolchains:

#!/bin/bash

set -x

sysroot="$(rustc --print sysroot)"
toolchain=$(basename "${sysroot}")
rustup target add powerpc64le-unknown-linux-gnu --toolchain "${toolchain}"
exec "rustup" "target" "list" "--toolchain" "${toolchain}"

I've made a simple script that tries to add a target and then list the available targets, the former which is absolutely necessary for cross functioning.

$ cargo-bisect-rustc --script=./list.sh -vv
installing nightly-2022-06-15
cargo for x86_64-unknown-linux-gnu: 6.56 MB / 6.56 MB [========================================================================================================================] 100.00 % 9.69 MB/s 
testing...
++ rustc --print sysroot
+ sysroot=/home/ahuszagh/.rustup/toolchains/bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu
++ basename /home/ahuszagh/.rustup/toolchains/bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu
+ toolchain=bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu
+ rustup target add powerpc64le-unknown-linux-gnu --toolchain bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu
error: toolchain 'bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu' does not support components: bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu is a custom toolchain
+ exec rustup target list --toolchain bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu
error: toolchain 'bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu' does not support components: bisector-nightly-2022-06-15-x86_64-unknown-linux-gnu is a custom toolchain

We can't add our target for cross-compilation to this toolchain, since it's custom. The reason why it works (spuriously) for me on nightly is because, and to quote their tutorial:

First, if you have a nightly version of the compiler already installed as the default toolchain and you don't pass an end flag, the tool is going to assume that that's the version that has regressed and use it as the "bad" version. Otherwise would use that start point or just use the latest nightly. If you have provided a start flag it would use that as the "good" version, otherwise is going to search for a good one backwards.

Look at cargo-bisect-rustc handles this, it seems to manually download components itself, which makes sense considering it can't use built-in rustup functionality for this:

https://github.com/rust-lang/cargo-bisect-rustc/blob/ab94c3b48e291f4bf100bb3edf4f273624918c17/src/toolchains.rs#L189-L195

So I guess the bigger question would be: can we support custom toolchains by adding the desired target in this case?