Doctests as part of travis

[vchuravy]

Doctests are part of the documentation and having them run automatically on travis, would ensure that the Julia documentation is kept up to date and consistent. Additionally more tests are always helpful and #15753 exposed a few regressions.

After #15753 all doctests are functional again and we could enable them for testing. While working on that PR I noticed that there is one annoying property of the doctest. They may spuriously fail when linenumbers change. This can be solved by replacing the linenumbers in the doctest with .... Any other changes will still trigger a failure.

As an example see 19d4431

[tkelman]

This will be yet another way in which travis can fail if people push changes straight to master without testing first, but I think it would be worth it.

For simple stuff like line numbers, I wonder if we couldn't make JuliaDoc (edit: or genstdlib) fill them in automatically? This may be easier for the doctests that have already been migrated to inline julia docstrings.

[yuyichao]

It'll be nice if the backtrace can be ignored .... As we improve the backtrace for inline functions the backtrace is likely going to be more and more sensitive to subtle changes that can affect line numbers or inlining decisions.

[vchuravy]

We probably could make JuliaDoc less sensitive to line-number changes, or alternativly we encourage the usage of .... As far as I can tell it is very sparsily used in the doctests. One example of where it is well used is: adce356#diff-c9107d78b307be1cab2363ee98efe06bR27.

[tkelman]

I actually like having a full detailed backtrace for release docs, but it is overkill while in dev.

[omus]

While working on #17106 I discovered some good reasons not to run doctests in dev (release may still be fine):

The doctest julia> methods(+) would sometimes print the methods in a different order resulting in the doctest to fail.

If we were to run the doctests as part of the Travis we would have to limit them to run on only 64-bit as integer literals reveal their types in exceptions :

julia> bar(a,b,x::Vararg{Any,2}) = (a,b,x);

julia> bar(1,2,3)
ERROR: MethodError: no method matching bar(::Int64, ::Int64, ::Int64)
...

I would however like to see some automation with the doctest system to ensure that the manual is kept up to date.

[KristofferC]

Closing in favor of #15815

[tkelman]

this is 15815

[KristofferC]

Derp, #19528.