Observable Notebooks 2.0 Technology Preview

by mbostock • 12 months ago

235 57 12 months ago

chaps4• 12 months ago

Wait is this different from canvas?

svieira• 12 months ago

Out of curiosity, why `<notebook>` as an element and not a standard web-component `<observable-notebook>` or something like that if the goal is to go all-in on "normal" web development?

pphysch• 12 months ago

I think the goal is explicitly not to go all-in on normal web development, to maintain some opportunity to monetize.

D3.js started as/still is just a (collection of) JS libraries. Then Observable came along, and while it is a nice tool for tinkering with D3, it was not at all obvious how to then move your finished Observable D3 viz to a fully self-hosted, regular website (without paying a subscription to use their custom runtime or servers or whatever).

Now I guess they realize they strayed way too far from "the web" and are back-tracking.

skybrian4• 12 months ago

They made the syntax look familiar so that standard editors (and LLM’s) will work, but this is still a custom file format that requires a build step to convert into an HTML page. See my other comment.

The build tool is open source and unlike a Jupyter notebook, it’s easy to edit without a custom editor. So you don’t need their editor, but it will still be nicer to edit with their editor.

drewda• 12 months ago

I was wondering the same.

FWIW, the HTML custom elements spec does require a hyphen: https://html.spec.whatwg.org/multipage/custom-elements.html#...

mbo• 12 months ago

self-plug that you might be interested in: I built a thin wrapper around the Observable Runtime that allows you to more easily use it in a pure HTML context: https://maxbo.me/celine/

ujkhsjkdhf234• 12 months ago

Observable's pricing is bit out of bounds for what they offer. It feels like a lot of these data startups are just looking for large companies that will unsuspectingly spend a ton of money on something they don't need.

gaws3• 12 months ago

Or get the golden ticket through an acquisition.

1dom• 12 months ago

I'm not sure I understand what the offering is here. I read the website, and watched the video, but it doesn't really make sense.

Is Notebook 2.0 just a HTML subset that starts with e.g. <notebook> instead of <html>?

Then what's the notebook kit npm package?

Then there's an editor?

I've come away from the page a little irritated, because starts with hyperbole like "the future of notebooks is here", and then the "hello world" in the new notebook 2.0 format looks like HTML, and a video of what could have just been a generic AI chat window.

Maybe I'm too hungry/tired to make sense of it right now. Will be interesting to read what others think.

chaps4• 12 months ago

Have a look at https://observablehq.com/@observablehq and just poke around.

It's a neat tool that solves a lot of specific sharing/visualization problems (think of newsroom data analysis).

I like observable but I hate how much it feels like the service has been taken over by product managers looking for that next bump.

skybrian4• 12 months ago

They probably could have explained better what hasn’t changed. Observable Notebooks work like a spreadsheet or makefile. The cells aren’t executed top to bottom (like a Jupyter Notebook). Instead they’re executed in dependency order.

Looks like this is still true with the new system:

The Build API analyzes cell source code to find unbound references, such that the appropriate variable graph can be initialized using the Observable Runtime. For non-JavaScript cells (such as Markdown, HTML, and SQL), the Build API also transpiles the source into a tagged template literal expression.

This is unlike other document formats. They made the cells look like standard JavaScript now (versus the JavaScript variant they had before) but the file format is necessarily different, because the execution model is different. There is a build step to ensure that the cells will be executed in the right order on page load.

jitl2• 12 months ago

I used the older online notebooks for a bit and really loved the immediacy but I’m always hurting for TypeScript syntax support. The is this a thing w 2.0? Something I’ll need to add later with Vite and only available locally?

It was a bunmer to not be able to copy paste code from my projects into observable. With local notebooks I see more potential for homegrown workflow but for me the benefit was always the platform and how starting a new thing was as simple as going to a webpage and clicking plus.

Anyways excited to play with this once it ends up in the online editor

mbostock2• 12 months ago

We’re interested in adding a TypeScript cell mode, probably first with type stripping rather than validating types. I was hoping for a pure JavaScript implementation; maybe we’ll try https://github.com/bloomberg/ts-blank-space? I’ve also been looking at https://github.com/val-town/codemirror-ts for the editor, but it’s a lot of machinery.

jitl2• 12 months ago

I would only expect type stripping, although it seems you already have the TypeScript language server hanging out in the editor for completion and docs and such, it’s not much extra challenge to get the VFS hooked up enough to show red squiggles in CodeMirror. I built a CodeMirror + typescript for the React docs site and it’s definitely machinery but not too bad maybe 600-800 LoC (never got merged, sad)

RobinL• 12 months ago

This looks great. I love the idea behind notebooks and for a long time it was my favourite environment to program in. But slowly I stopped using them because it never quite felt like the code was entirely mine, and alternatives became easier due to llms. This looks like exactly the remedy I was hoping for. I'm excited to start using them again.

kickout• 12 months ago

Big fan of the Notebooks. Hope the offline verison is hardened and available on windows soon enough.

These data viz are great dashboard builders but need a local/offline solution for my needs.

blef• 12 months ago

I'm curious to understand why did you decided to develop a local app?

jeffbee4• 12 months ago

I don't know but it's nice if the local app doesn't have the 50MB file size limit that the hosted notebooks have.

ModernMech• 12 months ago

It's pretty clear the future is not really here yet when the LLM can't even debug a simple table join issue on its own. The code to produce the table and chart should reasonably be like 2 lines of code with proper tool design. It looks like the LLM accomplishes it in 100 LOC and still can't manage to do it without significant human intervention.

And this is the take they published to sell the thing!

dr__mario• 12 months ago

I'd love to love them but I don't: I can't seem to justify moving away from Python to use this. What advantages does JS offer for this use case? I' ve never felt that I couldn't do any visualization with Python (but I don't do nice newspaper figures).

jeffbee4• 12 months ago

Maybe I just don't know enough python libraries but I can get to a good-looking interactive visual in Observable in almost no time at all.

MantisShrimp90• 12 months ago

Im in the same boat. In theory JavaScript holds more potential to make finely crafted visuals. But you're right, the ecosystem is so mature I still find other ecosystems lacking.

But if anything would change my opinion this has the right set of values

chaps4• 12 months ago

I'm not really a JavaScript person either, but that hasn't been the worst problem with observable. Tools like vega-lite and importable tools from other notebooks really reduces the amount of code I need to write. Like, I'm not writing d3 code for hours for a simple choropleth. What makes it easier is that I can query objects using SQL rather than lengthy JavaScript code.

wonger_• 12 months ago

Exporting as client-side webpages, maybe? I liked using it to prototype some interactive D3.js charts that I would later move to my website. Also some people just are more comfortable with JS than Python

skybrian4• 12 months ago

It depends where you're coming from, I suppose. The web is pretty popular. Web developers are more familiar with HTML, CSS, and JavaScript, and with various visualization libraries in the JavaScript ecosystem. I quite like Observable Plot [1].

I'm not familiar enough with Python or Jupyter to know how you would build similar visualizations with them. What would you use?

[1] https://observablehq.com/@observablehq/plot-gallery

ZeroCool2u• 12 months ago

You'd have to do some styling, but Plotly is what I'd use to recreate the graphs on the gallery page there. It would obviously require some work to match the exact styling, but it should all be doable and TTFP would probably be much lower assuming you're starting with a Plotly express template.

https://plotly.com/python/

Hasnep• 12 months ago

I think marimo is a python version of the same idea: https://marimo.io/

threecheese• 12 months ago

I am loving Marimo, I think they are doing a good job of balancing ease of use with the ability to drop into a virtualenv and run the code. The UI is much nicer than colab, from a visual and feature perspective. Key for me though is repeatability and ability to collaborate at the branch level, shared notebooks have issues we all know about.

What’s missing for me is a local desktop app like Jupyter Studio, but that’s an easy thing for the community to build.

nsonha• 12 months ago

never felt that I couldn't do any visualization with Python

Yes but, probably what they are shooting for are mini apps that you can just copy pasted into a front-end codebase and not some weird-ass python dsl

th0ma5• 12 months ago

They have or at least used to make their notebook format public, and I always thought if I ever see someone else adopt it then maybe the project will be worth looking at again. I think there could be better ways to capitalize on d3 but this one seems self limiting.

mritchie7122• 12 months ago

Observable has always demo'd well and looked cool, but I never understood the business application. Does anyone use it for work?

bsimpson• 12 months ago

I work on a quantitative user research team, and we've found notebooks to be really useful for crunching and visualizing the data from experiments we've run.

I work at Google and my UXR colleagues are more comfortable with Python than JS, so we use Google Colab, but I'd use Observable if those weren't true.

mritchie7122• 12 months ago

yeah, I know notebooks are used broadly (used them for years myself). I just haven't run into companies using Observable specifically in the wild.

jwilber• 12 months ago

By far the biggest complaint I hear from Observable users is the non-standard JS syntax, so great to see them moving to regular JS.

swyx2• 12 months ago

i dont understand why Observable insists on topological execution. isn't out of order execution one of the biggest complaints about notebooks?

jeffbee4• 12 months ago

Why should irrelevant nerd junk be at the top of my notebook? If the point of the notebook is to inform and educate the reader, that's a distracting limitation.

skybrian4• 12 months ago

Not exactly. Jupyter notebooks execute top-to-bottom at startup and then you can manually execute cells in whatever order you choose after that. I believe it’s the manual execution that gets people, because it can have inconsistent results. But just rerunning the entire file automatically (like reloading a web page) might be slow if some build steps are slow (they can read or write huge files), and some people have slow steps in their notebooks. They could use manual cell regeneration to intentionally avoid rebuilding cells, when stale data is acceptable.

On the other hand, I don’t know if Observable Notebooks 2.0 even does incremental builds anymore? If not, it’s just about presentation order, similar to literate programming.

Observable Framework had data loaders for creating static snapshots of data at build time. I don’t see any mention of how to handle data loading in Observable Notebooks 2.0.

gampleman• 12 months ago

Because it makes the reactive model work - and work quite efficiently. Cells don't re-run just because of edits, but also if the reader interacts with any of the interactive inputs.

lvl1552• 12 months ago

I wish Bostock et al got paid. This is literally the pain of doing open source for some. You basically watch as other people get rich off of your life’s work.

That rambling aside, I like the idea of having a desktop app. Not sure where this notebook fits in with all their other products. They have the Framework which is actually pretty useful.

chaps4• 12 months ago

Say more?

gaws3• 12 months ago

I wish Bostock et al got paid.

They do get paid. Observable sells team and enterprise licenses: https://observablehq.com/pricing

simantel• 12 months ago

Yeah Observable has raised $45M, I imagine the ELT is paid well.

jeffbee4• 12 months ago

Who parks a Porsche in front of a fancy Pacific Heights (San Francisco) mansion with the license plate "D3JS"?

lvl1552• 12 months ago

This is why I come to HN. Good for Bostock if true!

swyx2• 12 months ago

source pls?

mourner• 12 months ago

The updated syntax looks brilliant — can't wait to try it out by upgrading some of my older notebooks! I was worried about the future of Observable since the Canvas announcement — it felt like notebooks were being slowly phased out, but I'm glad to be proven wrong.

Any timeline on bringing the new format to the Web editor? For me, the biggest draw of Observable is being able to instantly start exploring and visualizing some data without ANY setup (creating some local files, installing dependencies, launching a dev server etc.) — just click "New notebook" and off you go, frictionless. I hope this remains a priority.

Also worth mentioning the data viz community aspect, which I used to enjoy a ton — it's a bummer that things like being able to follow certain users and see their new notebooks in a feed were phased out, and that trending notebooks are now hidden under the Help menu instead of being featured on the user's homepage. This might be one of the reasons I stopped coming back to the platform as often, although I realize building a community is very difficult and hard to monetize — just thought I'd share.

mbostock2• 12 months ago

Thank you. I don’t want to commit to a public timeline on the web editor yet, but rest assured it is a top priority, and we also value the convenience of a web editor, instant collaboration, and community!

simonw• 12 months ago

The Observable Desktop macOS app is a 11.8MB download which decompresses to a 23.9MB application. In a world filled with 200+MB Electron apps this is a very refreshing change!

stared• 12 months ago

It is interesting to see different formats for notebooks - Jupyter uses JSON, RMarkdown (as you may have guessed) Markdown, and Observable - HTML.

dunham• 12 months ago

I'd love to see support for inline TeX in the markdown. Back when I used observable on the web, I hacked up a `md2` string template to replace /\$(.*?)\$/g with the katex equivalent before calling `md`, but I don't know if there is a workaround in the desktop notebook application.

amdivia2• 12 months ago

Would this be open source? I would really like to run this locally

For the time being I hacked something using Observable Framework [1], but still, it isn't the full set

1: https://observablehq.com/framework/

Edit: nvm just saw the notebook kit

gaws3• 12 months ago

Would this be open source?

No? How would they make money?

amdivia2• 12 months ago

Part of it is open source (The part I care about)

Observable Notebook Kit | Observable https://observablehq.com/notebook-kit/kit

xyzzy_plugh• 12 months ago

This is really, really incredible stuff. I can't wait to replace a bunch of terrible BI dashboards.

I've seen and used so many amazing in-house dashboarding and reporting tools, and this suite blows them all out of the water. Can't believe this is all open source too.

I think this is a great way to slice up the product/feature set from a monetization perspective: hosting w/ collaboration is paid, but the tools are all free to use.

nixpulvis• 12 months ago

The video demonstrating the AI powered desktop app perfectly demonstrates my discomfort with AI. When asked to fix the mapping between county codes, it generated a lookup table itself.

HAVE FUN TESTING THAT. And when you don't, enjoy random countries being mixed up but it mostly working.

MrManatee• 12 months ago

Yeah. I can believe that this will improve in the future, and this is a technology preview and all. But my takeaway from the video is that it is still too vibey for me to trust it. If I was coaching a junior data analyst at my company, and they wrote this code, I would happily give some feedback so that they can improve. With AI, I'd probably just do the analysis myself.

In addition to the hardcoded lookup table, here are some other notes on the generated code:

1. Silently assuming that the first page of results contains all of the data feels a bit fragile. If we're not going to bother with paging otherwise, I'd at least assert that the first page contains everything.

2. Unlike the code comment claims, 2022 is not the latest year that has data available, 2023 is. The reason this is worrisome is not that the difference is massive, but because of methodological implications. It looks like the year 2022 came from "remembering" details from some other code that was analyzing this same data set instead of just looking at current version of the data set we're supposed to be analyzing.

3. The code for removing the world aggregate doesn't actually work (although it doesn't matter for the map). The place where it says d.country.id !== "WLD" should be either d.country.id !== "1W" or d.countryiso3code !== "WLD" instead. Also, if it would actually be important to filter this then presumably it would also be important to filter out a bunch of other aggregates as well.

4. The text says "darker colors indicating higher life expectancy", which is pretty much the opposite of how I would describe this color scheme.

5. The analysis given is: "Notice how certain regions tend to cluster together in terms of life expectancy, reflecting similar economic, healthcare, and social conditions". This is such a bland take. It feels like something I could have written before seeing the data. I would try to encourage the analyst to look for something more concrete and more interesting.

6. The actual thing that immediately pops out from the map is that the Central African Republic is a massive outlier with a life expectancy of 18.8 years, whereas every other country is over 50. This doesn't seem plausible. I would do something about that just so that it doesn't mess up the colors of the entire map.

kragen• 12 months ago

I've found Bostock Observable to be a fascinating and inspiring platform since the outset in my limited experiments with it. I wish they'd give it a name that isn't a well-known design pattern that it implements, because I end up having to call it "you know, Bostock's thing" in conversation.

However, I definitely don't want to have to type "<script id="2" type="module" pinned>" in order to calculate 1 + 2, and I don't want to have to scan past that in order to find the actual calculations. So I guess I'll have to keep being inspired by Bostock Observable Notebooks 1.0, which I can keep writing in GitHub-flavored Markdown, because it's free software and already checked into Git.

Observable Notebooks 2.0 Technology Preview

Related Stories