REPL mode for code blocks #1035

tlienart · 2023-07-09T20:13:59Z

ongoing todo list here: #1035 (comment)

Can now do

```>
x = 5
y = x * 2
```
some explanation
```>
z = 2y
```

and get the following html

<pre><code class="language-julia-repl">
julia> x &#61; 5
5
julia> y &#61; x * 2
10
</code></pre>

<p>some explanation</p>
<pre><code class="language-julia-repl">
julia> z &#61; 2y
20
</code></pre>

(note &#61 is =)

closes #818

cc @gdalle , @kescobo

gdalle · 2023-07-09T20:27:23Z

This is amazing! Thank you SO MUCH 😍 Testing as we speak
Our main use case for the blog post was illustrating the pkg, help and cmd modes, is that also part of this PR?

tlienart · 2023-07-09T20:27:53Z

one thing that could be polished is to store code chunks and results in files so that we don't have to re-eval if nothing changes (at the moment these blocks will be re-evaluated on every page change).

gdalle · 2023-07-09T20:34:59Z

one thing that could be polished is to store code chunks and results in files so that we don't have to re-eval if nothing changes (at the moment these blocks will be re-evaluated on every page change).

I spent hours this morning trying (and failing) to combine Replay.jl with asciinema to show things like

pkg> st

help?> map

in a reproducible way, so honestly rerunning everything upon page changes is the last of my worries 🤣

tlienart · 2023-07-09T20:39:20Z

Our main use case for the blog post was illustrating the pkg, help and cmd modes, is that also part of this PR?

no unfortunately, these would potentially also be a fair bit more difficult to do (to give a hint: you can't copy paste in the REPL something with a ? at the front and expect it to switch to help mode).

We could imagine having a ```?, ```] and ```; in a similar way as is done here but it'd require knowing how to run the relevant functions to retrieve the output of whatever the help mode calls etc. Definitely out of scope for the current PR. Your best option here is to just copy paste the REPL output into a julia-repl code block.

gdalle · 2023-07-09T20:40:36Z

Thanks for the explanation!

gdalle · 2023-07-09T20:43:41Z

The PR breaks on my macOS:

julia> using Franklin

julia> serve()
  Activating project at `~/Work/GitHub/ModernJuliaWorkflows`
→ Initial full pass...
┌ Franklin Warning: in <Project.toml>
│ Encountered an issue processing 'Project.toml' in ModernJuliaWorkflows.
│ Verify, then re-start the Franklin server.
│ The error is displayed below:
MethodError(Base._memcmp, (UInt8[0x5b, 0x64, 0x65, 0x70, 0x73, 0x5d, 0x0a, 0x46, 0x72, 0x61, 0x6e, 0x6b, 0x6c, 0x69, 0x6e, 0x20, 0x3d, 0x20, 0x22, 0x37, 0x31, 0x33, 0x63, 0x37, 0x35, 0x65, 0x66, 0x2d, 0x39, 0x66, 0x63, 0x39, 0x2d, 0x34, 0x62, 0x30, 0x35, 0x2d, 0x39, 0x34, 0x61, 0x39, 0x2d, 0x32, 0x31, 0x33, 0x33, 0x34, 0x30, 0x64, 0x61, 0x39, ...
└
ERROR: MethodError: no method matching _memcmp(::Vector{UInt8}, ::Vector{UInt8}, ::Int64)

Closest candidates are:
  _memcmp(::Union{Ptr{UInt8}, AbstractString}, ::Union{Ptr{UInt8}, AbstractString}, ::Int64)
   @ Base strings/string.jl:129
Stacktrace:
  [1] (::FranklinTemplates.var"#5#7"{IOStream})(file2::IOStream)
    @ FranklinTemplates ~/.julia/packages/FranklinTemplates/W5V3L/src/utils.jl:219
  [2] open(::FranklinTemplates.var"#5#7"{IOStream}, ::String, ::Vararg{String}; kwargs::@Kwargs{})
    @ Base ./io.jl:396
  [3] open(::FranklinTemplates.var"#5#7"{IOStream}, ::String, ::String)
    @ Base ./io.jl:393 [inlined]
  [4] (::FranklinTemplates.var"#4#6"{String})(file1::IOStream)
    @ FranklinTemplates ~/.julia/packages/FranklinTemplates/W5V3L/src/utils.jl:212 [inlined]
  [5] open(::FranklinTemplates.var"#4#6"{String}, ::String, ::Vararg{String}; kwargs::@Kwargs{})
    @ Base ./io.jl:396
  [6] open(f::Function, args::Vararg{Any})
    @ Base ./io.jl:393 [inlined]
  [7] filecmp(path1::String, path2::String)
    @ FranklinTemplates ~/.julia/packages/FranklinTemplates/W5V3L/src/utils.jl:211 [inlined]
  [8] process_file_err(case::Symbol, fpair::Pair{String, String}, head::String, pgfoot::String, foot::String, t::Float64)
    @ Franklin ~/.julia/packages/Franklin/Kqtgg/src/manager/file_utils.jl:169
  [9] process_file(::Symbol, ::Pair{String, String}, ::String, ::Vararg{Any})
    @ Franklin ~/.julia/packages/Franklin/Kqtgg/src/manager/file_utils.jl:104
 [10] fd_fullpass(watched_files::@NamedTuple{other::Dict{…}, infra::Dict{…}, md::Dict{…}, html::Dict{…}, literate::Dict{…}}, join_to_prepath::String)
    @ Franklin ~/.julia/packages/Franklin/Kqtgg/src/manager/franklin.jl:260
 [11] 
    @ Franklin ~/.julia/packages/Franklin/Kqtgg/src/manager/franklin.jl:124
 [12] serve()
    @ Franklin ~/.julia/packages/Franklin/Kqtgg/src/manager/franklin.jl:49
 [13] top-level scope
    @ REPL[6]:1
Some type information was truncated. Use `show(err)` to see complete types.

tlienart · 2023-07-10T06:55:47Z

This isn't related, to this PR. Which version of Julia are you using? the error seems to be caused by filecmp here: https://github.com/tlienart/FranklinTemplates.jl/blob/2e6534d36bfc3741203e1091b0e438450422cbd6/src/utils.jl#L197-L224 but that's weird.

Can you try serve(clear=true) and report (I just did, with a fork of your record_repl branch and it seemed to work fine)

gdalle · 2023-07-10T07:38:38Z

Thanks, it works indeed but there is a useless blank line before the first julia> prompt

Btw I'd love to add more contrast to the code blocks but I'm not sure how to change the background color within the template. Is it handled by the highlighting js?

tlienart · 2023-07-10T08:02:39Z

blank line should be removed

for the contrast, yes it's handled by highlight js; there are two things you could do here:

use a different style sheet (instead of the github one) using one of the style sheets in _libs/highlight/styles/ you can see demos here: https://highlightjs.org/static/demo/
manually customise the style sheet yourself

the style sheet are in _libs/highlight/styles/ and to change the ones you use you'd have to do that in _layout/head_highlight.html. For instance

<link rel="stylesheet" href="/libs/highlight/styles/atom-one-dark.min.css">

you could also use Documenter's one (https://github.com/JuliaDocs/Documenter.jl/tree/master/assets/html/themes)

gdalle · 2023-07-10T08:30:53Z

Thanks for the pointers! My goal is only to get a slightly darker background, shouldn't be too hard to achieve

tlienart · 2023-07-10T10:21:45Z

@gdalle the basic stuff is added, you can test the following:

```?
sqrt
```

or 

```]
st
```

or 

```;
echo "foo"
```

Note: for the help mode, Julia returns stuff in Markdown format; it can convert it directly to HTML and so here this HTML is shown inside a div container julia-help which would have to be styled. I haven't really played with this yet. Maybe have a go and lmk what you think? showing just plaintext won't work because there'll be stuff like ``` and [...](@ref) shown.

tlienart · 2023-07-10T10:38:55Z

additions to the docs

gdalle · 2023-07-10T11:41:27Z

This is awesome, thank you so much! It will improve the feeling of our blog post by 100. Testing now

gdalle · 2023-07-10T11:46:55Z

For the REPL mode, here are my first impressions

missing one line break between each prompt
nothing outputs should not be printed
errors are not shown?

Test code

a = rand(3, 2)
sum(a)
using LinearAlgebra  # this should print a void
1 + 1;  # this should print a void
# comment alone  # this should print a void
using Flux  # this should error
sum(exp(a))  # this should error

Result

gdalle · 2023-07-10T11:50:50Z

For the shell mode:

the prompt only appears on the first line
following lines are not interpreted
errors are not shown

Test code

echo "hello"
echo "goodbye"
echography

Result

gdalle · 2023-07-10T11:52:05Z

For the Pkg mode:

the prompt only appears on the first line
the name of the environment is not shown
adding a package fails
doing anything after adding a package also fails

Test code

st
add Random
st

Results

gdalle · 2023-07-10T12:00:40Z

For the docs mode:

I didn't manage to apply the styling (check out this commit to see if I did it wrong, I put the CSS in jtd.css since that's my style)
the prompt only appears on the first line
not sure what's displayed if we use a second input

Test code

Random
Int

Result

gdalle · 2023-07-10T12:04:51Z

And again, please don't take any of the previous messages as criticism. You asked for beta-testing so I'm gladly providing it :) I honestly didn't expect that my problem would be fixable, let alone so quickly, so I'm already beaming with joy and gratitude

tlienart · 2023-07-10T13:06:18Z

Thanks for the feedback @gdalle, some of these should be fixed in the current PR, some not. Notably capturing errors can be a bit annoying.

Separately: would you be open to moving to Xranklin for what you're building now? it might require a few adjustments but fixing some of the stuff you pointed out would be easier there (as I'm a bit reluctant to add a lot of code to Franklin 0.10)

stuff that can be done soon and easily

repl - adding lines between julia>
repl - nothing handling
shell - multi line, if we can make the assumption that each line is one Cmd
pkg - multi line

stuff that is likely to be hard(er)

help - multiline, likely too hard but I'm also a bit unsure of whether this is really an interesting use case
repl - errors (I need to try first, maybe this isn't too bad)
shell - errors (might be able to redirect stderr, would have to try first)
pkg - there's some work here (already done in Xranklin) to have the page have its own environment, there's some finicky stuff to figure out for the add/remove etc.
multi line shell stuff if we cannot make the assumption that each line is a single command --> probably too hard, won't happen (there'd need to be something similar to the repl mode where you can figure out whether the AST is complete or not)

gdalle · 2023-07-10T13:25:52Z

So should I keep using this PR or switch to Xranklin (perhaps on a specific branch?)

tlienart · 2023-07-10T13:26:53Z

(I wonder how Julia blog posts handle this actually)

I don't think any julia blog post uses evaluated code blocks

The repl items should all be fixed, including the error stuff (though it's not styled) so

```>
sqrt(-1)
```

for instance gives

<pre><code class="language-julia-repl">julia> sqrt&#40;-1&#41;
DomainError with -1.0:
sqrt will only return a complex result if called with a complex argument. Try sqrt(Complex(x)).

</code></pre>

tlienart · 2023-07-10T13:29:08Z

So should I keep using this PR or switch to Xranklin (perhaps on a specific branch?)

Maybe keep using this PR for now, I'll have a look at whether I can backport the Pkg stuff without making too much of a mess. All this stuff is not in Xranklin yet (but it's an ok draft).

shashi · 2023-07-10T19:21:27Z

@tlienart maybe it is easy to add an option to render the output using the richest MIME? Then we could have LaTeX output and Images!

Reminding you of
JuliaSymbolics/Symbolics.jl#527 which I never completed :)

tlienart · 2023-07-10T19:57:29Z

that's definitely out scope for the present PR, however on a more positive note, the situation is much improved in Xranklin which lets the user define their preferred "show" method for objects if you're interested see https://tlienart.github.io/Xranklin.jl/syntax/code_1/#output_of_executable_code_blocks. This already allows image stuff and should make LaTeX output a fair bit easier.

gdalle · 2023-07-11T06:08:10Z

Indeed the Julia mode looks perfect now!
Still haven't managed to style documentation output though, so it looks like plain text. My changes to the CSS were ineffective, could you maybe take a look at https://github.com/gdalle/ModernJuliaWorkflows/tree/repl and tell me what you think?

tlienart · 2023-07-11T07:34:14Z

@gdalle there's a stray } at the end of your jtd.css it should be

.franklin-toc {
  /* Avoid clickable elements being too close together. */
  margin: 0.6rem 0;
}


/* ==================================================================
    REPL BLOCKS
================================================================== */

.julia-help {
  background-color: yellow;
  padding: 10px;
  font-style: italic;
}

.julia-help h1,h2,h3 {
  font-size: 1em;
  font-weight: 500;
}

gdalle · 2023-07-11T07:53:45Z

@gdalle there's a stray } at the end of your jtd.css

Awesome, thanks!

gdalle · 2023-07-11T08:00:50Z

If we want to get the right colors for the pkg> (blue), shell> (red) and help?> (yellow) prompts, does it happen in the Julia code or in the style sheets?

tlienart · 2023-07-11T09:14:41Z

in jtd.css remove the hljs.metap, hljs.metas and replace them by

.hljs-meta.shell_ {color: red;}
.hljs-meta.pkg_ {color: cornflowerblue;}

(to have help mode being parsed in the same way you have to fiddle with _libs/highlight/highlight.min.js find the one for shell_ and copy it for help_)

tlienart · 2023-07-11T09:18:52Z

yeah so if in _libs/highlight/highlight.min.js you add

subLanguage:"julia"}},{className:"meta.help",begin:/^help\?>/,relevance:10,starts:{end:/^(?![ ]{6})/,

after

subLanguage:"julia"}},{className:"meta.shell",begin:/^shell>/,relevance:10,starts:{end:/^(?![ ]{6})/,

and then in jtd.css add

.hljs-meta.help_ {color: goldenrod;}

it'll get the styling right.

tlienart · 2023-07-11T09:20:47Z

Pkg mode is added (including multiline). Each page can activate its own environment now (recommended is to activate a temporary one). The site-wide environment is re-activated at the end of the page by default.

You can try

```]
activate --temp
add StableRNGs
```

then use it

```>
using StableRNGs
round(rand(StableRNG(1)), sigdigits=3)
```

I don't think I'll do the remaining items, so will let you play with this a bit, fix tests and stuff and eventually merge assuming it does what you expect.

gdalle · 2023-07-11T09:24:04Z

Amazing! I'll test right away

gdalle · 2023-07-11T09:40:09Z

That's really cool, the styling works (maybe we'll need a group PR to all Franklin templates?) and the multiline modes too. Only trouble is with the Pkg environment activation shenanigans:

When I don't do any activate in the code blocks, as soon as I interrupt the serve() process I land back in global environment @v1.9 instead of my package environment (MyBlog). I realized because I tried to relaunch serve() right away but of course @v1.9 does not know Franklin. It must be linked to what you do at the end to deactivate.
When I activate a temporary environment, it is retroactive and shows up in the previous prompts. Is it just esthetic (ie only the printed name) or are all the code blocks live-reevaluated in the temporary environment?

tlienart · 2023-07-11T10:10:45Z

both should be fixed now

gdalle · 2023-07-11T10:42:01Z

This looks really good to me, I don't have any other requests (lucky you 🤣).
We're gonna keep working on ModernJuliaWorkflows, in the short term how do you suggest we specify our dependencies? For now I depend on this specific tl/fr818b branch, do you plan on deleting it?

gdalle · 2023-07-11T11:26:23Z

Two minor details:

Spacing is a bit too narrow after the output of a help block
Would it be possible for said output to have round corners like the other blocks in Franklin?

I fixed it in my style sheet but it might be worth harmonizing in the future

tlienart · 2023-07-11T12:15:24Z

The styling stuff is up to the user; there could be demos in the franklintemplates which someone could pick up with adjusted CSS but for my part I'm going to skip on that in favour of working on Xranklin and upgrading the themes so they can work with it too. (also, you're likely to be the only one using this for a while).

I'll merge this in master as it mostly doesn't affect anything else and will do a patch-release with it (it's not what it is but I don't want to release 0.11 which I want to be the release with Xranklin).

thanks for your input, glad this helps!

gdalle · 2023-07-11T13:10:02Z

Sounds very reasonable, thank you again for your hard work! I'll make sure to mention you on the blog post

repl mode #818

8a91cb6

tlienart mentioned this pull request Jul 9, 2023

draft PR for fr818 #819

Closed

tlienart added 2 commits July 9, 2023 22:20

adding basic docs for auto/repl mode

bd556d6

small edits

bd85568

test fix

adad1f0

tlienart mentioned this pull request Jul 10, 2023

repl mode tlienart/Xranklin.jl#219

Closed

useless blank line

fb775b6

tlienart mentioned this pull request Jul 10, 2023

[Feature Request] Executed REPL block #818

Closed

rudimentary shell,pkg,help mode

8d9c4b8

tlienart added 2 commits July 10, 2023 12:25

small edits

7994dc0

adjustments + docs

7c20723

gdalle mentioned this pull request Jul 10, 2023

Recording interactive sessions modernjuliaworkflows/modernjuliaworkflows.github.io#8

Closed

tlienart added 2 commits July 11, 2023 09:46

multi-line shell

1fa7357

docs for shell mode

f645a5a

Pkg mode

ddfe174

removing terminal-specific code

43b425c

repl-help is only single line

37acd4e

fixes for pkg mode

10be954

gdalle mentioned this pull request Jul 11, 2023

Repl modernjuliaworkflows/modernjuliaworkflows.github.io#23

Merged

tlienart merged commit 6cae15e into master Jul 11, 2023
1 check passed

tlienart mentioned this pull request Jul 11, 2023

no method matching parse_fenced_block(::SubString{String}, ::Bool) #1037

Closed

gdalle mentioned this pull request Aug 4, 2023

What did this blog contribute to? modernjuliaworkflows/modernjuliaworkflows.github.io#46

Closed

REPL mode for code blocks #1035

REPL mode for code blocks #1035

Conversation

tlienart commented Jul 9, 2023 • edited Loading

gdalle commented Jul 9, 2023

tlienart commented Jul 9, 2023

gdalle commented Jul 9, 2023 • edited Loading

tlienart commented Jul 9, 2023

gdalle commented Jul 9, 2023 • edited Loading

gdalle commented Jul 9, 2023 • edited Loading

tlienart commented Jul 10, 2023 • edited Loading

gdalle commented Jul 10, 2023 • edited Loading

tlienart commented Jul 10, 2023 • edited Loading

gdalle commented Jul 10, 2023

tlienart commented Jul 10, 2023 • edited Loading

tlienart commented Jul 10, 2023 • edited Loading

gdalle commented Jul 10, 2023

gdalle commented Jul 10, 2023 • edited Loading

gdalle commented Jul 10, 2023

gdalle commented Jul 10, 2023

gdalle commented Jul 10, 2023

gdalle commented Jul 10, 2023

tlienart commented Jul 10, 2023 • edited Loading

stuff that can be done soon and easily

stuff that is likely to be hard(er)

gdalle commented Jul 10, 2023

tlienart commented Jul 10, 2023

tlienart commented Jul 10, 2023

shashi commented Jul 10, 2023

tlienart commented Jul 10, 2023 • edited Loading

gdalle commented Jul 11, 2023

tlienart commented Jul 11, 2023 • edited Loading

gdalle commented Jul 11, 2023

gdalle commented Jul 11, 2023

tlienart commented Jul 11, 2023 • edited Loading

tlienart commented Jul 11, 2023

tlienart commented Jul 11, 2023 • edited Loading

gdalle commented Jul 11, 2023

gdalle commented Jul 11, 2023 • edited Loading

tlienart commented Jul 11, 2023

gdalle commented Jul 11, 2023

gdalle commented Jul 11, 2023

tlienart commented Jul 11, 2023

gdalle commented Jul 11, 2023

tlienart commented Jul 9, 2023 •

edited

Loading

gdalle commented Jul 9, 2023 •

edited

Loading

gdalle commented Jul 9, 2023 •

edited

Loading

gdalle commented Jul 9, 2023 •

edited

Loading

tlienart commented Jul 10, 2023 •

edited

Loading

gdalle commented Jul 10, 2023 •

edited

Loading

tlienart commented Jul 10, 2023 •

edited

Loading

tlienart commented Jul 10, 2023 •

edited

Loading

tlienart commented Jul 10, 2023 •

edited

Loading

gdalle commented Jul 10, 2023 •

edited

Loading

tlienart commented Jul 10, 2023 •

edited

Loading

tlienart commented Jul 10, 2023 •

edited

Loading

tlienart commented Jul 11, 2023 •

edited

Loading

tlienart commented Jul 11, 2023 •

edited

Loading

tlienart commented Jul 11, 2023 •

edited

Loading

gdalle commented Jul 11, 2023 •

edited

Loading