Support multiple languages

[josevalim]

This is something we will definitely do at some point but not right now, since things are too much influx. This issue is therefore to collect interest.

Supporting other BEAM languages is easy and can be a stepping stone but other languages will require a communication protocol.

[adnikiforov]

I've managed to run Ruby code from a notebook, but I need to fix some things here and there, so no PR yet. But it kinda works. Same thing will work for Python in no time.

[TheGeorge]

Having Erlang support would be awesome! How would this integrate with the language servers to get stuff like auto-complete and similar?

[josevalim]

@TheGeorge we are not using language server here because we are using a much simpler variant. So whoever implements the Erlang bindings would need to implement the logic for autocompletion too.

[vans163]

.NET CORE C# (the one that runs inside a REPL on linux), Python would be awesome too.

[jkbbwr]

I have significant interest in using livebook to host "living unit tests"
That run specification and contract testing on a codebase via C.
We have the tooling ready to go, livebook would just need to invoke a shell process via ports maybe.

[filmor]

For my Erlang Jupyter kernel I worked on a generic jupyter library (https://github.com/filmor/ierl/tree/master/apps/jupyter, https://hex.pm/packages/jupyter) which should be relatively simple to extend to also implement a Jupyter client. That would open up support for basically the whole Jupyter ecosystem.

[josevalim]

After discussing with @jonatanklosko, we have realized it doesn't make sense for Livebook to support imperative/mutable languages, because the state tracking and cell evaluation mechanism rely on the assumption we can go back and re-execute an existing piece of code, which is hard on imperative languages since you can mutate any value created in a previous cell. So if we are branch out to the languages, we will likely continue to focus on functional ones.

[filmor]

It's not like Elixir or Erlang are really functional either (io:format("msg", []).).

[josevalim]

IO is not a good example here because it changes the state around the world in a way we can capture it and manage it.

In any case, that’s not the point here. Almost all languages will have observable side-effects. The question is how prominent those are in the code.

[filmor]

My point was that Beam languages don't even try that hard to be purely functional. There is ets, changing state in other processes, message passing, proc dict, etc.

[josevalim]

I am well aware. :) I didn't claim it is close to a purely functional one, but it is still a large departure from any language where every data-structure or every object is, by default, mutable.

[asmodehn]

I am wondering if Julia would stand a chance at working in an elixir livebook ?
I don't have direct experience with it but mutability seems to be explicit, it somehow targets data sciences, and there are some interesting math/science libraries to play with...
No idea how it would interface with the BEAM ecosystem though.

[mgibowski]

SQL

[josevalim]

Closing this for now. We do plan to support specific languages (such as SQL) but for now Livebook will likely remain specific to BEAM languages.

[michalmuskala]

Was there any progress on adding support for other BEAM languages? I'd be quite interested in helping out with adding Erlang support if this hasn't happened yet (and looking through the code I couldn't find any relevant references)

[josevalim]

Hi @michalmuskala, there was no progress here but we would love Erlang support.

For simplicity, we have decided that the Erlang runtime would still be implemented in Elixir, this way we can share most of the code and still use things like mix to install packages.

I believe the work is:

1. Add the ability to choose an Erlang cell in the UI (you can start with an additional Code button for now and we will decide the UI later). Make sure those cells are persistent as ```erlang in the markdown and properly read back
2. Implement evaluation for Erlang and exception printing
3. Implement the intellisense features for Erlang

There are some parts we haven't figure out, however:

1. Smart cells define variables, and they are written in Elixir, but you can't access Elixir variables from Erlang easily and you can't easily define Erlang variables from Elixir.

2. Erlang doesn't have an equivalent to Mix.install/2. We have two options here: keep the initial setup/installation in Elixir or we alias 'Elixir.Mix' to ?mix to allow Erlang developers to call Mix directly. Note however that smart cells have functionality to automatically add packages, such would need to be added to Erlang too if you go the second route.

Anything else I missed @jonatanklosko?

[jonatanklosko]

We also need Erlang syntax highlighting in the Monaco editor.

Given that the binding can't be shared I'm not sure if we should support Erlang as part of regular runtimes, or have a separate runtime Erlang instead. More specifically, I'm not convinced we should allow both Elixir and Erlang cells in the same notebook, because then it's less "convertable" to a single script. Dependency management is a good point though, but Erlang code that calls Mix sounds good, especially that we have UI for adding packages and the user doesn't need to be concerned with getting that code right.

[michalmuskala]

One option for sharing bindings would be to expose a map with them in each language: in Erlang this would be Elixir = #{a => var_a, ...} in Erlang it would be erlang = %{:A => VarA, ...}.

[filmor]

@michalmuskala Have you worked on this, yet? I'd like to have a look into this as well. For me, this would also be a lot more useful if Elixir and Erlang could be mixed in a notebook (using Mix.install and Explorer, but also being able to copy code in from the existing Erlang codebase).