Manually convert Pluto notebooks to .md

[gregoirepourtier]

• Fix crossrefs problem by avoiding to use @ref link or named @refs Crossrefs broken in Pluto notebooks #95
• try @maltezfaria's suggestion

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.04%. Comparing base (ffb260b) to head (eab7d9e).
Report is 11 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #96       +/-   ##
===========================================
- Coverage   79.86%   66.04%   -13.82%     
===========================================
  Files          25       24        -1     
  Lines        3025     3022        -3     
===========================================
- Hits         2416     1996      -420     
- Misses        609     1026      +417     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maltezfaria]

Here is something along the lines of what I had in mind:

using Inti
using Pluto
using JuliaFormatter

# from https://github.com/fonsp/Pluto.jl/pull/2471
function generate_plaintext(
    notebook,
    strmacrotrim::Union{String,Nothing} = nothing;
    header::Function = _ -> nothing,
    footer::Function = _ -> nothing,
    textcomment::Function = identity,
    codewrapper::Function,
)
    cell_strings = String[]
    header_content = header(notebook)
    isnothing(header_content) || push!(cell_strings, header_content)
    for cell_id in notebook.cell_order
        cell = notebook.cells_dict[cell_id]
        scode = strip(cell.code)
        (raw, ltrim, rtrim) = if isnothing(strmacrotrim)
            false, 0, 0
        elseif startswith(scode, string(strmacrotrim, '"'^3))
            true, length(strmacrotrim) + 3, 3
        elseif startswith(scode, string(strmacrotrim, '"'))
            true, length(strmacrotrim) + 1, 1
        else
            false, 0, 0
        end
        push!(
            cell_strings,
            if raw
                text = strip(
                    scode[nextind(scode, 1, ltrim):prevind(scode, end, rtrim)],
                    ['\n'],
                )
                ifelse(Pluto.is_disabled(cell), textcomment, identity)(text)
            else
                codewrapper(cell, Pluto.is_disabled(cell))
            end,
        )
    end
    footer_content = footer(notebook)
    isnothing(footer_content) || push!(cell_strings, footer_content)
    return join(cell_strings, "\n\n")
end

function generate_md(input; output = replace(input, ".jl" => ".md"))
    notebook = Pluto.load_notebook(input)
    header = _ -> "<!-- Generated by Pluto $(Pluto.PLUTO_VERSION) -->"
    fname = basename(input)
    function codewrapper(cell, _)
        # 1. Strips begin/end block
        # 2. Reformats code using JuliaFormatter
        # 3. Wraps all code in same ```@example``` block for documenter
        code = strip(cell.code)
        if startswith(code, "begin") && endswith(code, "end")
            code = strip(code[6:end-4])  # Remove "begin" and "end" and strip spaces
            # reformat code using JuliaFormatter
            code = format_text(String(code))
        end
        return string("```@example $fname\n", code, "\n```")
    end
    textcomment(text) = string("<!-- ", text, " -->")
    str = generate_plaintext(notebook, "md"; header, codewrapper, textcomment)
    open(output, "w") do io
        return write(io, str)
    end
    return output
end

## test it
dir = joinpath(Inti.PROJECT_ROOT, "docs", "src", "pluto-examples")
file = "poisson.jl"
file_in = joinpath(dir, file)
file_out = generate_md(file_in)

I think the code above generates the markdown we want. It has the advantage of not depending on anything other than Pluto itself (and JuliaFormatter, but that is optional). I am not sure which solution is best 🤔

[maltezfaria]

With the generate_md function above the documentation pipeline for Pluto notebooks would be:

1. Parse the .jl Pluto notebook files and convert them into a Documenter flavored .md
2. Let documenter run the file, tests, figures, etc, and create the .html

Maybe I am missing something, but this seems like it should work while being quite simple.

[gregoirepourtier]

I tried both pipelines, and IMHO the best solution would be to use your code to generate documentation from Pluto notebooks. The generate_md function requires much less pre-processing in the notebook itself and reduces the amount of post-processing tweaks needed. Also, it simplifies the process of getting cross-references to work properly.

So, I suggest we drop the PlutoStaticHTML.jl dependency and keep PlutoSliderServer.jl (along with ExampleJuggler.jl) to generate the HTML link for the notebook viewer.

[maltezfaria]

I tried both pipelines, and IMHO the best solution would be to use your code to generate documentation from Pluto notebooks. The generate_md function requires much less pre-processing in the notebook itself and reduces the amount of post-processing tweaks needed. Also, it simplifies the process of getting cross-references to work properly.

So, I suggest we drop the PlutoStaticHTML.jl dependency and keep PlutoSliderServer.jl (along with ExampleJuggler.jl) to generate the HTML link for the notebook viewer.

Thanks! I agree with you that the simple solution using generate_md is probably better, at least for now. I guess the last question is: do we really need the notebook viewer?

[gregoirepourtier]

If it works with the CI, we can consider adding it to replace the previous notebook viewer with the jupyter notebook.
For now, I’ll try without it to see if the rest works properly.

[maltezfaria]

Except for some broken links and minor formatting issues, this LGTM!

@gregoirepourtier Do you want me to take it from here and fix the remaining problems?

[maltezfaria]

We could also go ahead and merge this (probably rename the PR?), and I can start another one to fix the minor issues.

[gregoirepourtier]

The nbviewer is failing because of the following issue (and hence the broken links):

• showvalue not defined in Logging in Julia 1.11 julia-actions/GitHubActions.jl#36

If you see any other problems, please go ahead and fix them, thanks! And also yes to the renaming and start a new PR