False positive: baseline rustdoc built from registry version only includes default features

[obi1kenobi]

When cargo-semver-checks constructs a baseline rustdoc JSON file, it (implicitly) activates the default features only. The "current" rustdoc is instead constructed with --all-features.

If a feature e.g. adds an enum variant, that variant is detected as being newly-added: exists in current, does not in baseline. This causes false-positive semver errors.

Example affected crate: libp2p-core v0.37.0

$ cargo semver-checks check-release -p libp2p-core
    Updating index
     Parsing libp2p-core v0.37.0 (current)
     Parsing libp2p-core 0.37.0 (baseline)
    Checking libp2p-core v0.37.0 -> v0.37.0 (no change)
   Completed [   1.094s] 18 checks; 17 passed, 1 failed, 0 skipped

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-check/tree/v0.12.0/src/queries/enum_variant_added.ron

Failed in:
  variant PublicKey:Rsa in core/src/identity.rs:223
  variant PublicKey:Secp256k1 in core/src/identity.rs:226
  variant PublicKey:Ecdsa in core/src/identity.rs:229
  variant PublicKey:Rsa in core/src/identity.rs:223
  variant PublicKey:Secp256k1 in core/src/identity.rs:226
  variant PublicKey:Ecdsa in core/src/identity.rs:229
  variant Keypair:Rsa in core/src/identity.rs:73
  variant Keypair:Secp256k1 in core/src/identity.rs:76
  variant Keypair:Ecdsa in core/src/identity.rs:79
       Final [   1.175s] semver requires new major version: 1 major and 0 minor checks failed

The equivalent call using --baseline-rev reports no errors: cargo semver-checks check-release -p libp2p-core --baseline-rev libp2p-core-v0.37.0

Additional info:

• Enum where features add variants: https://github.com/libp2p/rust-libp2p/blob/bcff814b92c76706a2df46098e3c8a61f29ef9fc/core/src/identity.rs#L216-L230
• Affected PR: .github/workflows: Enforce semver compliance with cargo semver-checks libp2p/rust-libp2p#2647
• Details: .github/workflows: Enforce semver compliance with cargo semver-checks libp2p/rust-libp2p#2647 (comment)
• Log of the false-positive error: https://github.com/libp2p/rust-libp2p/actions/runs/3254423615/jobs/5342659716

[obi1kenobi]

@epage is loading the list of features from the index and adding it to the dependency line a good idea?

Or is it better to try to match the crate version to a tag in the git repo using a set of heuristics like v{version} or {crate_name}-{version}?

I don't fully understand the code around there — especially how we manage to get a crate's rustdoc by making a new manifest by declaring a dependency on it — so I'd love your take on how to go about this.

[epage]

@epage is loading the list of features from the index and adding it to the dependency line a good idea?

Ideally we'd centralize our handling of features (RustDocCommand does --all-features iirc) but absent that, then yes, that would be the best way to handle that.

especially how we manage to get a crate's rustdoc by making a new manifest by declaring a dependency on it — so I'd love your take on how to go about this.

When we run rustdoc, we pass in <name>@<version> to request one specific crate's documentation from the dependency tree, the dependency.

[obi1kenobi]

Ideally we'd centralize our handling of features (RustDocCommand does --all-features iirc) but absent that, then yes, that would be the best way to handle that.

I think that --all-features ends up only applying to the root crate (named rustdoc) and not to the dependencies.

I was able to confirm by hand that the following sequence works:

• update the index and generate a lockfile with only default features for the crate
• use cargo metadata to get the full list of features for the crate
• explicitly add all features to the dependency line in Cargo.toml
• delete the old Cargo.lock to avoid a resolution failure somehow caused by the old lockfile: .github/workflows: Enforce semver compliance with cargo semver-checks libp2p/rust-libp2p#2647 (comment)
• generate the rustdoc as before

That's my plan for moving forward since it seems to work well.