Add a rustdoc flag to show the source code externally

[pietroalbini]

One of the features of docs.rs is to show the source code in its UI, but rustdoc provides its own view of the source as well. This duplication is both confusing for the user, but also duplicates the amount of storage docs.rs uses to store source code.

It'd be nice to have a rustdoc flag (unstable is fine) like --source-code-base, that prevents rustdoc from emitting the source pages on its own and replaces the [src] links with links to the source code base. For example, with --source-code-base ../../crate/lazy_static/1.4.0/source, the [src] for lib.rs would be ../../crate/lazy_static/1.4.0/source/src/lib.rs, and rustdoc's own src page would be missing.

cc @GuillaumeGomez @jyn514

[Mark-Simulacrum]

Have we discussed doing the reverse? That is, pointing docs.rs generated sources into rustdoc [src] sources?

[pietroalbini]

Well, as with everything it's a tradeoff. While reducing the amount of code in docs.rs is tempting, I think the advantages of serving it from docs.rs are worth more. @rust-lang/docs-rs, what do you think?

Pros for serving the source code from rustdoc:

• Less code for the docs.rs team to maintain.
• The UX is the same as local docs.

Pros for serving the source code from docs.rs:

• Less storage space used, as docs.rs stores the raw source and renders it on the fly.
• We can update the interface without having to rebuild the world.
• We can guarantee the source code was not changed by a build script.

[Mark-Simulacrum]

If we're not storing rendered source code anyway then there's not really duplicate storage implications? (Just wasted storage from rustdoc, which we could plausibly redirect ourselves, without rustdoc changes?)

We could probably decompress crates.io tarballs on the fly and pull out the appropriate file we need.

[pietroalbini]

At the moment, for each source file, we're storing:

• rustdoc's html file with the pre-rendered highlighted source code
• the original source file (in s3://rust-docs-rs/sources/...), smaller as it doesn't have HTML in it

We should stop storing one of the twos.

We could probably decompress crates.io tarballs on the fly and pull out the appropriate file we need.

That could slow down the server a lot, as we have a few crates that go over 10MB.

[Mark-Simulacrum]

It seems like linking to rustdoc's pre-rendered sources is the best (and plausibly easiest) bet, personally.

That could slow down the server a lot, as we have a few crates that go over 10MB.

Yeah, that's true. More of a loose thought -- maybe e.g. it makes sense to do so for small crates.

[jyn514]

At the moment, for each source file, we're storing rustdoc's html file with the pre-rendered highlighted source code

It's actually even worse than this, we store a separate copy of rustdoc's source code for each platform. See e.g. https://docs.rs/regex/1.3.1/i686-pc-windows-msvc/src/regex/lib.rs.html and https://docs.rs/regex/1.3.1/i686-unknown-linux-gnu/src/regex/lib.rs.html.

[jyn514]

We can update the interface without having to rebuild the world.

This is a very convincing argument (to me) for not using the rustdoc sources. I think I've linked more to rust-lang/docs.rs#464 than to any other issue.

[jyn514]

Btw, I don't think it would be too hard to implement this on the docs.rs side without adding a new flag to rustdoc. We could just redirect /:crate/:version[/:platform]/src/:crate-name to /crate/:crate/:version/source/src/

[GuillaumeGomez]

Adding an option to rustdoc to link to a given location would be easy and seem to be what we want to do in here. That'd remove the duplicated files. When everyone agrees on a solution, I'll implement it.

Note for myself: if this option is used, no need to generate source code files.

[jyn514]

Adding an option to rustdoc to link to a given location would be easy.

I'd be happy with this, but it only works with future docs. What if we did both: rustdoc adds this option which links directly and then docs.rs still redirects past crates to the version of the source code we host? That would allow getting rid of a lot of storage costs for all those crates and also fix issues like rust-lang/docs.rs#484 and rust-lang/docs.rs#359

[camelid]

We can update the interface without having to rebuild the world.

This is a very convincing argument (to me) for not using the rustdoc sources. I think I've linked more to rust-lang/docs.rs#464 than to any other issue.

IIUC, you'd only be able to update the source code view, not any other parts of the docs. In my experience, the source code view is rarely the thing I wish were up to date when viewing old docs. I don't really understand why it's valuable to make the source code view—but not the rest of the docs—easy to update.

Also, updating just the source view of old docs would make their style inconsistent with the rest of the docs, which could be annoying for users as well.

[jsha]

This seems like the wrong direction to go. Compare https://docs.rs/crate/rustls/0.20.2/source/src/builder.rs vs https://docs.rs/rustls/0.20.2/src/rustls/builder.rs.html#97-100.

The rustdoc-generated source view is more featureful:

• It has line numbers (extremely useful for linking someone to a specific line or range of lines).
• The file navigation allows toggling open directories without doing a full page navigation.
• @GuillaumeGomez's "jump to definition" feature is only available via the rustdoc-generated view.
• It uses the same syntax highlighting colors that the docs pages do for examples.

Users will still see the rustdoc-generated source view in many other places: doc.rust-lang.org, plus self-hosted docs and local filesystem docs. It doesn't make sense to ask them to learn a different UI for source view on docs.rs.

If the goal is to reduce conceptual duplication (a good idea!), it makes sense to remove https://docs.rs/crate/rustls/0.20.2/source/ in favor of the rustdoc-generatd view. For the separate goal of saving space, there are ways to achieve it while still using the rustdoc-generated source view. For instance, docs.rs could omit the src/ directory from its doc bundles, and generate a separate platform-independent doc bundle that contains the src/ directory from the default platform's build.

[jyn514]

The rustdoc-generated source view is more featureful

If the goal is to reduce conceptual duplication (a good idea!), it makes sense to remove https://docs.rs/crate/rustls/0.20.2/source/ in favor of the rustdoc-generatd view.

The problem is each has features the other doesn't; in particular rustdoc only shows files in the library itself, it's missing Cargo.toml and build.rs and anything else that doesn't directly have documentation generated.

I agree changing docs.rs to only store src/ for one platform makes sense.

[jsha]

Ah, I forgot to address that aspect of things. Yes, I think it's important that there be an authoritative way to view the contents of a specific crate artifact, including non-.rs files. But that should probably be on crates.io.

(Alternately, rustdoc could implement that feature too)

[jyn514]

Well, you can already use cargo download. But it's nice to be able to browse it online, I use that feature quite a lot to read Cargo.toml versions and examples.

[GuillaumeGomez]

It would be interesting for rustdoc to generate source code pages for all files included in the released crate I guess. I'm also not a big fan of having sources in two different places. We could argue that the display of docs.rs-specific sources is a bit better than rustdoc's and I'd agree. However, I expect it to be fixed by #89385.

[camelid]

For instance, docs.rs could omit the src/ directory from its doc bundles, and generate a separate platform-independent doc bundle that contains the src/ directory from the default platform's build.

Could this break go-to-definition for platform-specific code though?

[jyn514]

It could, yes.