kvlr

Hey! We're excited to finally open source Clerk. Clerk is a local-first notebook, built on top of regular Clojure namespaces.

It can take a namespace like https://github.com/nextjournal/clerk/blob/fd79f7504ebffaa279... and turn it into https://storage.googleapis.com/nextjournal-snapshots/clerk/b...

The main idea behind Clerk is letting folks use the editors they know & love and not force them to edit code inside a web browser. (This came to us only after working for a few years on nextjournal.com where folks can edit notebooks in a web browser but better late than never :)

In order to keep the feedback loop fast Clerk keeps a dependency graph of all forms and vars in a notebook to figure out what changed between runs on a notebook and only runs those.

There's no homepage for it yet but Clerk is running a handful of sample notebooks on every commit and building a static website out of them. The latest version is at https://storage.googleapis.com/nextjournal-snapshots/clerk/b...

Hey, I’m a founder of Nextjournal and one of the authors of clojure-mode.

Highly recommend checking out CodeMirror 6 [1], it’s really well done. On of the main benefits is its blazingly fast incremental parsing system called lezer [2], which is heavily influenced by tree-sitter. The Clojure grammar file is at [3].

I’ve done a small video about this at launch [4] and you can also play with this on Nextjournal [5].

Let me know if you have questions or feedback.

[1] https://codemirror.net/6/

[2] https://lezer.codemirror.net/

[3] https://github.com/lezer-parser/clojure/blob/master/src/cloj...

[4] https://twitter.com/mkvlr/status/1352274315354529792

[5] https://nextjournal.com/try/mk/rule-30

Here’s a runnable version of this book https://nextjournal.com/try/plfa/Naturals

For our notebook format at Nextjournal we use an edn appendix to store metadata like results and environment (docker image). The user-edited content is just plain markdown, while also being runnable reproducibly. See [1] and run it at [2].

[1] https://github.com/nextjournal/notebook-format-demo/blob/mas...

[2] https://github.nextjournal.com/nextjournal/notebook-format-d...

Hey HN, I'm one of the founders of Nextjournal. The `cm6=1` query param is a feature flag to enable our new Clojure code editor built on CodeMirror 6 [1] with lezer for fast incremental parsing. You can find the keybindings on [2], be sure to try grow / shrink selection.

[1] https://github.com/nextjournal/codemirror.next-clojure

[2] https://nextjournal.github.io/codemirror.next-clojure

I used mmhmm today to record a quick demo [1] and think it works really well for that. I like the looks a lot more than the normal rectangular overlays in Zoom or ScreenFlow.

[1] https://twitter.com/mkvlr/status/1331630391233564676

This is how runkit does it for nodejs and I think it’s working quite well for them.

We (at Nextjournal) tried doing the same for other languages (Python, Julia, R) and felt that it didn’t work nearly as well. You often want to change a cell at the top e.g. to add a new import and it can be quite annoying when long-running dependent cells re-execute automatically. I now think that automatic execution of dependent cells works great when your use case is fast executing cells (see observablehq) but we need to figure out something else for longer running cells. One idea that I haven’t tried yet is only run cells automatically that have executed within a given threshold.

I hear a lot of complaints about hidden state but I think it’s less of a problem in reality. It’s just a lot faster than always rerunning things from a clean slate. Clojure's live programming model [1] works incredibly well by giving the user full control over what should be evaluated. But Clojure's focus on immutability also makes this work really well. I rarely run into issues where I'm still depending on a var that's been removed and then there's still the reloaded workflow [2].

Overall I think notebooks are currently a great improvement for people that would otherwise create plain scripts – working on it is a lot quicker when you have an easy way to just execute parts of it. Plus there's the obvious benefit of interleaving prose and results. That doesn't mean we should not be thinking about addressing the hidden state problem but I think notebooks do add a lot of value nevertheless.

[1] https://clojure.org/guides/repl/introduction

[2] http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-r...

Author here, I did run it a few times but you don’t have to take my word for it, you can rerun the notebook yourself if you sign up for Nextjournal and remix my notebook. Full disclosure: I’m also a Cofounder of Nextjournal.

But I wouldn’t suggest to use Nextjournal for serious benchmarking (yet). We’re running on Google Cloud and it’s not suited for benchmarking unless you pay for a (big) sole tenant instance. In the future we plan to offer dedicated instances for benchmarking.

Can't say if it does but at https://nextjournal.com we do. We also support running on GPUs.

TLDR: MongoDB

Ah nice, we do want to switch Nextjournal to Shenandoah eventually after reading http://clojure-goes-fast.com/blog/shenandoah-in-production/

So is graal polyglot already compatible with newer JDKs and it's just native image generation that's blocking the upgrade? https://github.com/oracle/graal/issues/651

What you're describing is close to our long-term vision for the platform. Having it be easy to use for non programmers but with the ability to customise things and always "peek under the hood" to see how things are built.

Our GitHub and S3/GCS Components (see it in action in my launch post) are actually just thin layers that execute code from other notebooks and we plan to offer this ability to create custom components like this in the future.

I haven't tried Domino Datalab myself so take this with a grain of salt.

While data science is an obvious use case of literate programming, it's not the only one. I see the fundamental problem that needs to be addressed is one of dependency management. We address this today using Docker. In the future we plan to use a more functional approach most likely based on Nix or Guix. This more principled approach should address both reproducibility and usability (by allowing to compose images and providing much better install times thanks to binary caching).

I haven't really used Domino Datalab but I'm not sure if they allow for the installation of arbitrary system libraries and packages like we do. Check out some out our machine learning samples which run on GPUs: https://nextjournal.com/collection/machine-learning

In the future we also plan to allow in-browser JavaScript execution, this is currently hidden behind a feature flag but we still have an article that uses it in https://nextjournal.com/dubroy/ohm-parsing-made-easy

Yes, we certainly want to support a lot more languages in the future and it should be relatively easy to do as long as there's a Jupyter kernel (which is a lot).

We also have a proof-of-concept PR where you can implement a runtime in a notebook.

I think we'll expand the available languages as soon as we're confident the core product really solid.

Our standard instances at 3,75 GB of Ram and we keep a pool of three idle ones of those around. With the free account you can currently use larger instances of up to 16 GB of Ram and 1 Nvidia K80 GPU for free.

If you sign up for the paid plan which is 99$ per researcher per month you can provision more powerful machines – basically anything that Google Cloud offers.

We currently don't enforce any storage limits.

This is our first iteration of pricing though so I'm pretty sure this will still change over time. We've gotten a lot of feedback from people asking for a cheaper plan.

What most people don't realise however is that you can use most of the features (including private drafts) as it stands now for free. We've also been debating weather we should allow for private drafts on the free plan or take a stance on what open science really means (working in the open from the start) but decided agains this for now.

Curious to hear what others think about this. Do you expect drafts to be private and would it be a violation of those expectations if they were not?

oh and I was able to reproduce the comment issue and will look into it!

Awesome, happy to see that. Been wanting to play with crux anyway. Nextjournal runs on Clojure and Datomic and we use some of pack.alpha and aero from juxt, so thanks!

Hey, this sounds like something we'd want to look at, I don't think it's inherent to reproducibility. I think we should be truncating the output in this case. Please send us the notebook via the Help button that shows up when you produce an error in a cell, I'll take a look and we'll figure this out.

While Colaboratory is built on top of Jupyter Nextjournal is not.

We do support importing Jupyter notebooks and running Jupyter kernels, we also have our own runtime protocol.

In Jupyter (and hence in Colaboratory) you normally have one runtime that's running both your server code as well as the user code. In Nextjournal there's a separate application called the Runner that's orchestrating the runtimes which currently are docker images.

This allows us to use Nextjournal notebooks to do any kind of installations without the need for a full Jupyter kernel inside the image, something that gets tricky in Jupyter. Once we have a bash shell inside the image, we can do installations.

You can choose to commit the filesystem state at any time as a docker image and reuse it in other notebooks. This is actually how our default environment images are built: Our default Python environment https://nextjournal.com/nextjournal/python-environment is built on top of the minimal bash environment https://nextjournal.com/nextjournal/bash-environment which is importing just a stock ubuntu image.

Our system takes care of only referencing the image sha's everywhere, so everything is immutable and you can't accidentally overwrite anything.

You can also pull those docker images and use them locally.

Any data you upload or results you save (just write to a /results folder) is put into content-addressed storage, so same thing here, you'll never accidentally overwrite a file.

Lastly the document is stored in the database (Datomic) and you can restore any previous state.

Leveraging immutability at all layers of the stack is what enables our "remix" feature, so the ability to quickly and cheaply clone any published notebook and continue where another person left off.

for reference: https://nextjournal.com/r3tex/loss-landscape is the article he's talking about.

While I can't say anything definitive or give a timeline we do want to support research groups like yours. Ideally we'd be able to have our paid offering for companies using Nextjournal in private subsidize our open science/source offering.

We also definitely want to open source parts of our product but we haven't figured out what parts (or everything) and under what license.

Our priority is currently on providing a useful hosted product and become sustainable. It's certainly also interesting to see how e.g. metabase is doing it the other way around, open source first without a hosted product but I guess I'm a bit scared of not being ready for developing Nextjournal in the open at this point in terms of bandwidth and keeping things backwards compatible.

We currently don't enforce any limits on data storage and the largest singe file I know of is 110 GB. Ideally, we'd like our paid version to subsidize the open source/science version similar to how it works on GitHub.

I’m one of the founders of Nextjournal and I’m really excited that after almost three years in Private Beta we’re finally opening signups to everyone today!

Nextjournal is a computational notebook platform and our goal is to make computation more accessible and automatically reproducible, so it becomes easier to collaborate and build on top of each others work.

If you'd like to know more, check out our launch blog post at https://nextjournal.com/mk/public-beta or sign up and give it a try!

I’m one of the founders. The environments notebook includes the default environments you get when starting a notebook on Nextjournal. These are again built in Notebooks, see e.g. for https://nextjournal.com/nextjournal/python-environment for a description of what our default Python environment contains and how it is built. This is again built on top of our bash environment, that’s built on top of a stock ubuntu image: https://nextjournal.com/nextjournal/bash-environment.

It's not. Replaced it with a 600kb JPEG version, thanks!

Yes, we're based in Berlin. Should put the link to https://nextjournal.com/imprint back into the footer.

You can read about how Sauspiel – a community for the German card game Schafkopf – allows us to build Nextjournal on https://nextjournal.com/about.

Sorry about the glitch. Can you give it another try? You can use `xoxo` as an invite code for the private beta if it works this time.

Thanks, fixed.

Right, we didn’t get around to splitting the js bundles up yet but will do so soon, thanks for the reminder. Currently the main bundle is the same for the page and the editor which you can try at https://nextjournal.com/try

Not yet. It isn't based on Jupyter but it's all written in Clojure. Been meaning to do a writeup on our stack for some time…

In addition to our own runtime protocol, we support Jupyter kernels and you can import Jupyter and (R)markdown documents.