Livebook: A collaborative and interactive code notebook for Elixir

https://dashbit.co/blog/announcing-livebook?new=1

by bcardarella • 5 years ago

585 85 5 years ago

So this is like Jupyter notebooks for Elixir?

rajandatta• 5 years ago

Yes - it would seem so. The support for custom run times is interesting and seems like an improvement. Not sure Jupyter notebooks do that - certainly not for all supported kernels.

nerdponx• 5 years ago

Jupyter is great but it's somewhat of a "lowest common denominator". So there is always the temptation to make a more task-specific version; see also the Pluto notebook project for Julia and R Markdown (although the latter evolved concurrently with IPython/Jupyter, not after it).

LSP is in a similar situation right now, where it's great for languages that lacked tooling to begin with (like Python), but it's not yet possible to migrate existing powerful language tooling (Lisp SLIME, OCaml Merlin) to use LSP.

lawik5• 5 years ago

With google docs style collaborative editing, can connect to an existing cluster to run code in it. You can run one instance of the Livebook application and have different livebook docs run for different Elixir projects you happen to be working on.

Not sure what parts are in Jupyter but I've heard collab editing isn't.

Pretty hype about trying this for some living docs and teaching. Will see if I can do anything neat with it. Already tried it some.

bostonvaulter22• 5 years ago

Yeah I think this will be really neat when paired with an example project for libraries.

aleDsz2• 5 years ago

I used to teach my viewers of the basics of Elixir. It's in Brazilian Portuguese, but it's kinda interesting how easy is to use Livebook and show some code for people.

https://www.twitch.tv/videos/991797943

matreyes• 5 years ago

If you compare the work done in both projects (github insights), you will feel the power of the Beam, Elixir, Phoenix and LiveView.

dnautics4• 5 years ago

"connect to an existing node" is going to be amazing.

I imagine two interesting uses for this.

1. producing reports. Connect a livebook session to your system in prod, run a livebook (which pulls data from DB eg), then get back a report. User engagement, cloud resources consumed, p90s, or an incident report...

2. manual database intervention. Suppose you need to manually make a batch of changes to the DB. It would be very cool if you could run a livebook inside of a sandboxed checkout (a temporary transaction) and write your db changes, then write your code that verifies that the db is in a state that you want it to be in. If you mess up your code, then you just terminate the transaction without committing. Then when you're happy with your notebook, you run it once with a commit at the end, and then save your notebook immutably in your records so that you can keep track of what you've done.

I think for intervening in prod, it would be very cool if someone could figure out how to extend erlang distribution over websockets so you don't have to figure out complicated mechanisms (open up ssh port, forward encrypted epmd, e.g.) to get into your system in prod

qbasic_forever3• 5 years ago

Be careful, websockets are tricky to secure and get right. It's easy to do something like forget to check the origin and now you've opened up a whole class of cross-site websocket scripting attacks. I love the idea of easy access and nice UI to prod machines, but put it behind a secure proxy or VPN layer and not facing the public internet.

dnautics4• 5 years ago

That's very reasonable.

bcardarella• 5 years ago

Phoenix Channels accounts for many of these attack vectors. Livebook uses Phoenix LiveView which is built on Phoenix Channels.

wrren• 5 years ago

'Connect to an existing node' is likely implemented using ERTS (http://erlang.org/doc/reference_manual/distributed.html). So you're basically connecting to the runtime of the target node. The security model for ERTS isn't great, though, so your point stands. All you need is a matching secure cookie and you can connect and run any commands you want.

exikyut• 5 years ago

Hmmm.

Nginx has an auth_subrequest module that yeets all the cookies in a requests over to a configured URL in a server-initiated subrequest (hence the name), then proceeds with the original request only if the subrequest returns 200 OK. The module works with any type of incoming request, including FastCGI and reverse proxy situations.

The idea is that some config magic turns the 401 into a 302 redirect that walks you through a single-signon page, then you get redirected back to wherever you left off. However since you can make failed requests do just about anything you could technically reroute the request to a WebSocket server that just serves error responses or something, but that feels very... wronky.

The module does feel somewhat unwieldy to use in general as well, although this may be due to general unfamiliarity on my part with Nginx's way of doing things (which I'd summarize as "AAAAAAaaaaaaaa (aaa!)"). (The whole `if`-statements-can-SEGV thing etc.)

The module can only return 200 (allow original request through) or 401 (deny original request); any other response causes nginx to cancel the original request and return 503. (In case of 401, anything else, or error, the FastCGI server or upstream never sees a connection at all.)

It's possible to pass interesting state information such as a username or non-opaque session token from the subrequest to the main request by capturing subrequest response headers then injecting them as request headers into the main request, but this is sufficiently into the weeds (and unlike any existing system Web developers would typically have experience with) that I have a strong tendency to just go \ N O \, but considering that it is admittedly an obscure yet established corner of nginx's configuration, I figured I'd say something for completeness... for the benefit of the 2 people still reading up to this point. :D

I've incidentally been trying to figure out how to set up a simple BeyondCorp-esque system with it for a while now (for remote access to home systems, so focusing mostly focusing on brain-dead simplicity), but the giant brick wall I keep running into is how to a) make things auth-aware, b) keep the cookie properly opaque, and b) prevent everything from exploding horribly if auth_subrequest gets turned off. The only method I can think of to enable (b) (so that individual services don't have to duplicate auth logic, kind of defeats the point) is to pass headers out from the subrequest, which then breaks (c) (making it possible to access things directly).

A small microservice that handles the cookie side of things and makes HTTP requests itself starts looking like a breath of fresh air...

nonbirithm• 5 years ago

I wish nREPL would be standardized to be non-Clojure-specific as with LSP, so you could connect to a program written in any language with an nREPL server and poke at it. This is basically the Elixir version of that.

devoutsalsa• 5 years ago

We use Elixir at work. I attach to a running sapp all the time. It’s great!

mstipetic2• 5 years ago

These guys are amazingly productive, while also running a commercial company

fastball3• 5 years ago

This is software that directly helps their business, so it's not really productivity outside of their business endeavors.

xcambar2• 5 years ago

Try not to sound like you're diminishing the efforts and accomplishments of others.

This is tremendous work, they're running their business and come up with generic/generalisable solutions instead of narrowing the outcome to their own benefits only.

Pandabob• 5 years ago

Is it just me or is Elixir being talked/written about more in the past couple of months?

cpursley2• 5 years ago

Years.

cpursley2• 5 years ago

Interesting that I'm getting down voted, I'm long on Elixir and betting my startup on it.

SatvikBeri• 5 years ago

There were 300 elixir stories in 2020, an average of roughly .8 per day. In 2021 there have been 89 elixir stories for 108 days, so about the same. (Source: hnsearch)

bostonvaulter22• 5 years ago

Is it possible to tell how many of those hit the front page? I think the majority of those 300 stories in 2020 did not hit the front page. I'm curious how many there were in years prior as well.

lawik5• 5 years ago

It is growing and having good adoption. Not a huge language but it also has a fair share of novel features (being built on Erlang) that aren't commonly seen with other langs. So I think it might have a disproportionate visibility compared to size. Lots of work happening in the community, lots of ambitious ideas and releases.

joelbluminator9• 5 years ago

Why do you say its growing?

cultofmetatron2• 5 years ago

A bunch of startups including my own bet big on elixir 2 years ago and are starting to launch and gain prominence. the first wave of early adopters are coming out with battle stories to share and the conclusion is that elixir is the real deal.

bmitc• 5 years ago

... the conclusion is that elixir is the real deal

This makes me even more excited to start an upcoming new position writing Elixir full-time!

out_of_protocol• 5 years ago

Readable `.livemd` format looks lovely (regular markdown without extra elements)

http://lon-screenshots.s3.amazonaws.com/2021-04-18_Screensho...

AlchemistCamp• 5 years ago

I'll admit that I was initially skeptical about Elixir for anything ML related when I first picked up the pragprog book on Elixir and Genetic Algorithms.

Since that time, things have moved quite a bit. Between the Erlang VM JIT, then Nx, and now this, it's starting to look hopeful!

I'm looking forward to digging into this.

CTmystery• 5 years ago

Awesome work! Towards the end, José hits on a couple points that Jupyter does not do well:

  - Minute 25: Saved notebook is readable (markdown)
  - Minute 23: Live collaborate on a single notebook
  - Minute 22: Visual indication that cell is stale after an upstream cell changes

srush• 5 years ago

I've been playing around with some of these ideas for Python notebooks.

https://github.com/srush/streambook

This is a proof of concept that combines Jupytext for markdown readable notebooks with Streamlit for in-order execution+caching. More difficult to get some of the more advanced features of Pluto / Livebook due to Python state management.

Curious to hear any thoughts.

O_H_E• 5 years ago

Would you be able to expand or drop a few keywords on how (and what is) "python state management" prohibit few advance features.

By in-order-execution you mean pluto's reactivity, right.

transfire• 5 years ago

All this cool Elixir news. I'm really having a hard time deciding between Elixir and Julia.

(For programs that I must have compiled to a distributable executable, I've pretty much settled on Crystal. I find it more productive and pleasent than Go or Rust.)

thruflo22• 5 years ago

It’s not quite the same story but https://github.com/spawnfest/bakeware builds elixir releases into distributable executables.

lobo_tuerto• 5 years ago

That one is out of date, the one it forked of is not: https://github.com/bake-bake-bake/bakeware

cultofmetatron2• 5 years ago

if you want to build a saas, go with elixir. Its got a great batteries included web framework thats built to scale and an emerging suit of ml and linear algebra software coming out.

If you want to experiment with ML and scientific programming where you do a lot of matrix processing, go with julia. thats literally what its made for.

neolog2• 5 years ago

Julia has a web framework too of course. https://genieframework.com/

neolog2• 5 years ago

https://github.com/JuliaLang/PackageCompiler.jl

dnautics4• 5 years ago

As someone who started on julia first and then moved to elixir:

If you're doing computational/numerical stuff, stick with julia, for now. You really can't do that elixir, not today, and maybe not for at least another year or two. It will likely never be as good as julia since computation is not first class in the vm it runs on.

If you're doing or planning or doing anything with web, or orchestration, go with elixir.

namelosw• 5 years ago

Wow, this is great.

The fact is can connect to existing nodes also makes it can also be great for operation and administration - instead of building an awkward admin panel, you can just having a collection of scripts.

The Elixir community is insanely productive. Also, the Elixir system is impressively pretty balanced - productivity-wise like Rails, concurrent-wise like Erlang, and continuously brings novel yet practical ideas to life in a robust manner!

cancan• 5 years ago

Love how prolific the elixir community and especially José is. Also, if you are into liveview/elixir, definitely dive into the codebase. There is a small operational-transform lib built in there too.

losvedir2• 5 years ago

I'm pretty excited about this. I'm admit I was skeptical when I saw a similar LiveView based notebook a couple months ago, and wondered why not just make a kernel for Jupyter.

But with the idea of real-time collaboration I get it. And with the momentum of Jose and team behind it, I think there's real promise here, especially once they figure out widgets and charts, which they have an open issue for.

I program in Elixir at work, so I usually have an iex shell open, but lately I've been using a LiveBook and it's great.

And I love the new Mix.install in Elixir 1.12 demonstrated here. Using a hex package in an adhoc way was a real shortcoming in Elixir before. Shows the power of having this thing developed by the language team.

filmor• 5 years ago

There are already Jupyter kernels, IElixir and ierl (which I wrote). The latter supports Erlang, Elixir and LFE right now and supports connecting to another node as well.

https://github.com/filmor/ierl

hades32• 5 years ago

Super nice! It's only missing auto-complete and it'll be light-years ahead of other tools on terms of usability

chrismccord• 5 years ago

There’s already a PR for it with a video of autocomplete in action :)

https://github.com/elixir-nx/livebook/pull/208

losvedir2• 5 years ago

I'm just gobsmacked at how fast they're moving here. This is incredible.

spicyramen• 5 years ago

If I'm a data scientists with a bunch of existing Python notebooks can I use this product?

astrowilson5• 5 years ago

Not yet. It only supports Elixir for now (there is an issue for supporting other languages) and the notebook formats are different, so someone would also need to write a nbconvert to Markdown (.livemd).

ch4s3• 5 years ago

I'm not sure how I'm going to use this yet but I think there are going to be some great opportunities for prototyping here.

aleDsz2• 5 years ago

I'm using this tool to teach the basics of Elixir on my livestream. I think there's a lot of good usecases for Livebook, not only machine learning and stuff like that.

rkss• 5 years ago

Would like to know how Livebook achieve collaborate-editing. Does it use algorithms like CRDT or OT ?

lawik5• 5 years ago

OT. Using quill js on the frontend and a subset of that delta format handled on the backend. From conversation with Jose on Twitter.

cocco2• 5 years ago

Using it everyday. Great tool.

cocco2• 5 years ago

Using it everyfay. Greate tool.

arthurcolle• 5 years ago

How to use dependencies within a livebook?

I just tried to use the enum_type (version 1.1.3) by just installing it inside the cloned livebook repo, but I'm getting the error that EnumType isn't loaded.

Why is this?

Livebook: A collaborative and interactive code notebook for Elixir

Related Stories