Documentation Documentation Documentation

[MikeInnes]

Here's a first pass at writing some meta-documentation for the documentation system. Documentation.

I'll hold off on merging this until the type stuff actually, you know, works, and open it up for feedback in the meantime. Let me know if there's anything you're unsure about that this doesn't clear up.

[MichaelHatherly]

I'm just beginning to work on this for Docile. If everything works out right then the 0.3/0.4 documentation transition should be quite smooth.

[timholy]

This is great to have. Mike & Michael, you have really changed the landscape!

[MichaelHatherly]

Thanks @timholy! Along with all the other big changes in 0.4 (such as #7941), the landscape will certainly be changing significantly.

[StefanKarpinski]

+1e9 – I can't express how important and greatly appreciated your work on this has been. This is going to be one of the bigger deals in 0.4.

[dpsanders]

Hmm yes, sorry, I think my word choices were too strong. And I commit agree: the documentation tools are really fabulous!

[ViralBShah]

Thanks to all this amazing work, we are likely to have great package documentation, which is a big gap in the Julia ecosystem today.

[quinnj]

Is Juno mentioned anywhere else in the manual? I don't mind including it, but we may want to include more info about it and other relevant commands perhaps.

[timholy]

Or just a link to Juno's home page.

[timholy]

Is it possible to document whole modules? I'm thinking along the lines of

help?>Optim
# Displays the README.md for Optim, if nothing more specific has been defined

[MikeInnes]

Not yet, but that's a really good idea. Feel free to open an issue to remind me.

[timholy]

Bump. I see a couple of easily-actionable items in the comments, can those be done and then merge this?

[timholy]

I was pleased to discover that this works:

for (f, op) in ((:add, :+), (:subtract, :-), (:multiply, :*), (:divide, :/))
    @eval begin
        $f(a,b) = $op(a,b)
    end
end
@doc "`add(a,b)` adds `a` and `b` together" ->add
@doc "`subtract(a,b)` subtracts `b` from `a`" ->subtract

Feel free to copy/paste this demo in.

In general, I'd say this PR only covers the surface of the system, and a somewhat more extensive treatment would be helpful. But this is a great start.

[MikeInnes]

Glad to see people excited about this! I've tried to address the feedback/spelling mistakes etc. (apologies for taking so long on this) and in the interest of getting this out there I'll go ahead and merge. If I've missed any of the feedback here or anything else could be clearer, please do open an issue and ping me.

[timholy]

Yay! Thanks, @one-more-minute.

[ViralBShah]

Fantastic! Looking forward to everyone transitioning to the new docs now.

[hayd]

When should an effort begin to migrate Base? Do we first need to have the manual populated from @doc (so that documentation isn't duplicated) or is this already set up?

[ViralBShah]

That is the first step. We also then need to hook up help(), which is the easy part. I wonder if someone has a script that does the migration from the current format to the markdown format we want. We should probably also strive to structure each entry a bit better (R does a good job), and there was discussion on this in another issue, which I will dig up.

[jiahao]

So is there an accepted syntax for cross-referencing other functions/types in the new documentation format? There's been quite a lot of effort put in to make the cross-references; it would be a shame to let that go to waste.

[hayd]

Currently works with REPL help mode and Base.@help but not with Base.help ?

Structure discussion was here: #8966. +1 for cross referencing/related functions as a standard section (as discussed in that issue).

This PR notes

Documentation is very free-form; there are no set formatting restrictions or strict conventions. It's hoped that best practices will emerge fairly naturally as package developers learn to use the new system.

IME best practices rarely emerge naturally, would be great to come up with an example(s) of a "good" doc.

[jiahao]

@hayd I'm concerned that the migration from the current ReST format will be lossy, since the current stdlib docs are quite extensively cross-referenced.

Cross-references as a section like "See also:" is insufficient. Most of the current cross-references are inline.

[jiahao]

IME best practices rarely emerge naturally, would be great to come up with an example(s) of a "good" doc.

Have you seen the help for functions like eachindex, lufact, eigs or fft recently? The Base docs already use nontrivial features of Sphinx/ReST for tables, executable code snippets (extractable and runnable as doctests), citations, cross-references and equations. Not all of these features exist in standard Markdown. Equations we could try to finagle with injected LaTeX like IPython notebook does, but I haven't seen a discussion of cross-referencing syntax yet.

Not to denigrate the work put into the @doc macro so far, but we need support for all these features at the very least before even considering the migration of the standard library docs.

[ViralBShah]

These are all valid points.

@MichaelHatherly @one-more-minute Should we track the various features we need before we can migrate as a separate issue?

[MichaelHatherly]

Yeah, I tried some time ago to convert some docs from rst to md using pandoc and it wasn't really a 1-1 conversion. @jiahao's definitely right. A new issue tracking needed features would probably be worth doing.

[MikeInnes]

I think it's completely fair to focus on package documentation for now – that's the much more urgent need, nice though it would be to have colourful docs for Base.

@jiahao is right that Base is a bit more complicated, though the hard part is mainly decisions like where documentation should live and whether we want to build to RST etc. We have #8967 open where we can discuss those decisions, if anyone has any opinions on them.

Things like references are pretty much trivial to implement, e.g.

type Reference
  obj
end

ref(x) = Reference(x)

writemime(io, "text/restructured", ::Reference) = # ... whatever the appropriate syntax is ...

doc"See also $(ref(fft))"

We could also just specialise this to

doc"See also [fft]"

but I'd need to think some more about how that plays nicely with links.

[hayd]

@one-more-minute I wonder if a nice syntax would be doc"See also $[fft]". Edit as will conflict with latex.

Does this have to be something packages can define/overwrite - so that their references link to the correct page ?

[MikeInnes]

The Reference type itself could certainly be used by other packages, etc., but it would only make sense when outputting to formats like RST that support references directly. I'm not sure if you could do other formats like HTML generically.

[tshort]

Mkdocs supports cross references, tables, and Latex equations (with a plugin). References are normal markdown links. Here are two examples:

See [IdealDiode](../../lib/electrical/#idealdiode) and [Event](#event).

One references a function cited in another file, and one references a link in the same file. That might not be as smart as the Rst approach, but it gets the job done.

Mkdocs could be used for the online docs, and features could be incrementally added to Julia's markdown support to improve live HTML.

[hayd]

I wonder if this should/could be done with @MichaelHatherly's Lexicon. It would be pretty awesome if the docs for the language were generated (from code) in exactly the same way as packages (with the same automated tool).

@jiahao I have a PR for MD tables, so that's part of the way there :). Executable code snippets could just be the same as Code objects, but labelled as such... Perhaps we need a tracking issue for all the features needed but we don't yet have?

As discussed before, some kind of MetaDoc so you can pass a sequence of labelled sections would be really nice - this would also avoid inconsistent standards for headers in docstrings (# vs ## vs ###).

[MichaelHatherly]

@hayd, I think it comes down to what should be in base and what shouldn't, ie. #5155. Interesting enough, with Docile's new plain docstrings (without @doc) it could be possible to have docstrings without adding anything to Base. Not sure how others feel about that though.

[hayd]

@MichaelHatherly IMO being included in base is independent to being required/used to build the docs, I'm not advocating Lexicon be included in base! (Edited for clarity.)

[MichaelHatherly]

@hayd, thanks for clarifying that.