Tracking issue: rfc 1990 - add external doc attribute to rustc

[nrc]

RFC PR: rust-lang/rfcs#1990
RFC text: https://github.com/rust-lang/rfcs/blob/master/text/1990-external-doc-attribute.md

Current documentation

---

Summary of how to use this:

1. Add #![feature(external_doc)] to your crate.
2. Add a file "src/some-docs.md" with some docs to your crate.
3. Add an attribute #[doc(include = "some-docs.md")] to something that needs some docs. The file path is relative to lib.rs, so if you want a doc folder to live alongside src, then all your paths inside the doc(include) attributes need to begin with ../doc.

Summary of current status:

• Initial implementation landed in rustdoc: include external files in documentation (RFC 1990) #44781, on 2017-11-22.
• Updates based on conversation in this thread landed in tweaks and fixes for doc(include) #46858, on 2017-12-21.
• Please use this in your own projects and comment in this thread if something goes wrong! (Or even if it goes well! Knowing it's working as intended is great!)

---

Current tasks:

• Land rustdoc: include external files in documentation (RFC 1990) #44781
• (promote the "failed to load file" warnings to proper lints?) (Make them hard errors instead, per the RFC tweaks and fixes for doc(include) #46858)
• Ensure line number info is properly preserved in doctest errors

[matklad]

cc @joshtriplett

At today's Cargo meeting, and interesting question was raised how Cargo should check if it should rebuild the docs when an external file with docs changes.

Here's some info on how dependency-tracking works today:

When doing cargo build, Cargo passes an --emit=deb-info flag to rustc, which causes it to emit dep-info files with .d extension, which lists what files were used for compilation (remember, only crate roots are explicitly mentioned in Cargo.toml). These files then used in fingerprint.rs to compute the fingerprint.

Curiously, cargo doc does not emit dep-info at all, and fingerpint.rs does not use depinfo when doing doc. Instead it asks Source for fingerprint which for the local repository boils down to listing all files in a directory with Cargo.toml.

TL;DR: touch Readme.md today causes cargo doc to rebuild docs for the current crate, which might not be the most efficient thing to do, but which should work with external documentation, unless it is fetched from a place outside of Cargo package.

[joshtriplett]

@matklad So, it sounds like the current behavior is correct but could be made more efficient. That's better than being incorrect.

[matklad]

So, it sounds like the current behavior is correct but could be made more efficient.

There's an edge case where it might not do rebuild when it should, if in src/lib.rs you have something like

#[path="../../bar.rs"]
pub mod bar;

Or

include!("../../bar.rs")

That is, if you touch files outside of the Cargo package directory.

Other than that yes, current behavior is correct!.

[luser]

This probably won't cause any issues in the real world, but related to the discussion above about dependency tracking, I noticed that #[doc(include="file")] doesn't wind up in the dependency info output by rustc --emit=dep-info, which doesn't seem right. (In practice, since doc comments presumably don't affect the compiler output, it probably wouldn't cause any actual correctness problems).

Compare vs. include_str! for example:

luser@eye7:/tmp$ rustc +nightly --version
rustc 1.24.0-nightly (560a5da9f 2017-11-27)
luser@eye7:/tmp$ cat > test1.rs <<EOF
> #[allow(unused)]
> const s: &str = include_str!("other-file");
> EOF
luser@eye7:/tmp$ cat > other-file <<EOF
> hello!
> EOF
luser@eye7:/tmp$ rustc +nightly test1.rs --emit=dep-info -o test1.d
luser@eye7:/tmp$ cat test1.d
test1.d: test1.rs other-file

test1.rs:
other-file:
luser@eye7:/tmp$ cat > test2.rs <<EOF
> #![feature(external_doc)]
> #![doc(include="other-file")]
> EOF
luser@eye7:/tmp$ rustc +nightly test2.rs --emit=dep-info -o test2.d
cluser@eye7:/tmp$ cat test2.d
test2.d: test2.rs

test2.rs:

It looks like include_str! et. al. simply call cx.codemap().new_filemap_and_lines(&filename, &contents) to make this happen:

rust/src/libsyntax/ext/source_util.rs

Line 152 in a0db04b

// Add this input file to the code map to make it available as

so presumably doing the same thing inside the implementation of #[doc(include="..")] would fix this (probably here, I'm guessing):

rust/src/libsyntax/ext/expand.rs

Line 1117 in 632ad19

let include_info = vec![

[luser]

I neglected to mention, but it looks like there's even a test to ensure that files used in include_str! and include_bytes! wind up in the dep-info: https://github.com/rust-lang/rust/tree/b1363a73ede57ae595f3a1be2bb75d308ba4f7f6/src/test/run-make/include_bytes_deps

[QuietMisdreavus]

Humorously enough, i asked about that when i was writing it:

<misdreavus> hmm, should i add files loaded by doc(include) into the codemap? include_str does
<jseyfried> I would lean against

Judging from the above discussion in this thread, it may be prudent to add it, but it also may not be necessary. I'd defer to @rust-lang/compiler for that, i guess.

[joshtriplett]

I personally would find it quite unfortunate if changing the included file doesn't lead to rebuilding the code. (One day, incremental compilation should make that a very fast rebuild.) I realize that usually it won't affect the generated code, but I could imagine a syntax extension of some kind processing doc comments (after all documentation has been normalized to the equivalent of a doc attribute containing a string), and it shouldn't be the job of that syntax extension to handle dependencies on included files.

[jonhoo]

The current implementation seems to interact poorly with #![deny_missing]:

$ cat test.md
foo
$ cat test.rs
#![doc(include = "test.md")]
#![feature(external_doc)]
#![deny(missing_docs)]

fn main() {}
$ rustc +nightly test.rs
error: missing documentation for crate
 --> test.rs:1:1
  |
1 | / #![doc(include = "test.md")]
2 | | #![feature(external_doc)]
3 | | #![deny(missing_docs)]
4 | |
5 | | fn main() {}
  | |____________^
  |
note: lint level defined here
 --> test.rs:3:9
  |
3 | #![deny(missing_docs)]
  |         ^^^^^^^^^^^^

error: aborting due to previous error

[petrochenkov]

@jyn514 is planning to move this forward (to removal) by getting rid of existing uses of doc(include) (#78835), so unassigning myself.

[trevyn]

For reference, #[doc = include_str!("my_doc.md")] is now stable in 1.50 via-- #78837

Edit: Ooh, how embarrassing. Still behind #![feature(extended_key_value_attributes)] feature gate. (tracking #78835) Thanks @jyn514

[jyn514]

@trevyn include_str is not stable, it's still behind a feature gate.

[camelid]

To be clear, include_str! is stable, #[doc = include_str!] is not.

EDIT: clarified

[jyn514]

@jyn514 is planning to move this forward (to removal) by getting rid of existing uses of doc(include) (#78835)

I opened #82539 deprecating doc(include). Assuming that makes it past FCP, I'll give it a few months then remove the feature altogether.

[gilescope]

That relative paths niggle: Has anyone written a macro that fixes the relative links?
#[doc = deepen_relatives!(include_str("../README.md"), 1)]