Jupyter Notebook 5.0

http://blog.jupyter.org/2017/04/04/jupyter-notebook-5-0/

by sornars • 9 years ago

588 110 9 years ago

blog.jupyter.org

FiReaNG3L• 9 years ago

The only thing I want from Jupyter notebook is easy version control workflow - not sure what's the best option is right now but last time I checked it was 'delete all outputs before commit' which is great in many cases.

RobinL• 9 years ago

There's some good advice here: http://stackoverflow.com/questions/18734739/using-ipython-no.... Obviously imperfect, but it helps until it's implemented properly

Y_Y• 9 years ago

Use nbstripout[0] as a git filter. Then you have seamless git control of notebooks. There's talk of automatically saving code only separate versions in future releases.

[0] https://github.com/kynan/nbstripout

lancefisher• 9 years ago

This looks pretty good for handling diffing/merging of notebooks. I'm just getting started with jupyter, but it seems like somewhat of a pain to have to ssh into a server to manage versioning with git while I work on the code in the browser. I'd prefer if I could do all of my jupyter work directly in a notebook, and commit from the notebook webpage.

I'm not sure how others work with this stack, so there is likely tooling I don't know about. I'd love to hear suggestions.

spott2• 9 years ago

Check out nbdime[0], it might be what you are looking for.

[0]https://nbdime.readthedocs.io/en/latest/vcs.html

spott2• 9 years ago

This probably needs more explanation:

From the docs: "nbdime provides tools for diffing and merging Jupyter notebooks."

It includes both graphical diff and merge tools, command line diff and merge tools, VCS integration for git (so git uses the nbdime diff and merge for notebooks), etc.

Might be worth looking into if that is what you are looking for.

goerz• 9 years ago

I would even settle for an undo/redo (of more than one action)

roystonvassey4• 9 years ago

Jupyter will become like the Google of software development then ;)

To be fair, wasn't Jupyter designed to be an IDE - experiment with code, make tweaks and get the final version out in a text editor - rather than a repository of production-ready code?

carreau11• 9 years ago

Not really, you rarely have IDE designed to be around live-code. IDE are more meant to develop full-featured software. We are moving in this direction (as many users request it), but we still want to stay focus on Data Exploration.

Sean1708• 9 years ago

IDE here meaning Interactive Development Environment, rather than Integrated Development Environment?

m_mueller2• 9 years ago

This very much. I'd like for Jupyter to have an option to keep the cell code in individual python files in th backend, so you can just check in those to VCS. It would furthermore allow outside python code to import code from notebooks if jupyter would just provide a simple __init__.py file together with the cell files.

carreau11• 9 years ago

So you are asking for https://github.com/takluyver/nbexplode and https://github.com/ipython/ipynb respectively. There just not maintained enough to be in the core :-)

mctx• 9 years ago

You can create a post save hook to output .py files - http://jupyter-notebook.readthedocs.io/en/latest/extending/s...

(You used to be able to run ipython notebook --script)

TimSAstro• 9 years ago

This is something I've been giving some thought to recently. I came to the conclusion that the best way to go currently is git-clean filters built using jq. It's still a bit hacky though. Gentle write-up here: http://timstaley.co.uk/posts/making-git-and-jupyter-notebook...

roystonvassey4• 9 years ago

Hands down the best and most used tool in my portfolio. Well done, guys!

Does anyone know if there's a plan to introduce multi-kernel support in single notebooks like what Zeppelin does? Not that I have a strong preference for it but it appears to hold a lot of appeal in Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark.

nl• 9 years ago

Spark-like environments where not all packages are available in Pyspark and you need to move between native Scala/spark and Pyspark.

What exactly do you mean here? Are you referring to the parts of Spark which don't currently have a Python API? Because those are becoming smaller and smaller.

There is also Jupyter magics to let you change languages within a notebook. See %Rpush and %RPull from [1]. Not sure if there is a way to have a Scala kernel running and sharing the same Spark context though.

I think IBM is working on something in this area.

[1] https://blog.dominodatalab.com/lesser-known-ways-of-using-no...

roystonvassey4• 9 years ago

Yes, that's what I exactly meant. And you're right - that's a pretty small subset and that's what I tell others too.

But, the ease with which you can load interpreters on Zeppelin (apart from the pre-loaded ones) is impressive. I imagine it comes at a cost of some instability because it hangs more often than Jupyter.

carreau11• 9 years ago

See project like metakernel [1], in the end a kernel can be "just" controlling process that manage multiple language. So why you won't get "multiple kernel" you can get "one kernel with multiple language". Its just asking for "multiple kernel" which not the best way to describe what you want to do. In the end it the same excpet you do the language dispatch in the backend instead of the frontend. It's also more efficient for data sharing.

[2] show you haw Hydrogen (based on Jupyter as well), does it.

And [3] (mine), show you how in the the same notebook to use Python, R, C, Rust, Fortran, Cython and Julia with data sharing and sending functions back and forth between languages. I was definitively lazy and did not include things like SQL, javascript and a few others. I haven't used spark in a while, and definitively never from scala directly, but I doubt it would be much harder to do as the C/Rust/Fortran/Cython took me an afternoon to write.

[1] https://github.com/Calysto/metakernel [2]: https://www.google.com/url?hl=en&q=https://medium.com/nterac... [3]: http://carreau.github.io/posts/23-Cross-Language-Integration...

dmix• 9 years ago

I'm curious, what do you use Notebook for on a frequent basis? Data analysis?

roystonvassey4• 9 years ago

Primarily, yes. And also for building and trying out ML models.

spot• 9 years ago

We are actually working on improving the polyglot support in Jupyter by porting/merging work from Beaker Notebook (http://beakernotebook.com) into Jupyter and especially into Jupyter Lab. This new effort is called BeakerX: https://github.com/twosigma/beakerx. We have autotranslation working in one direction now, should have multi-way in a couple of weeks.

cercatrova2• 9 years ago

Like Beaker Notebooks?

zeptomu2• 9 years ago

Did anybody succeed in deploying Jupyter into a multi-user environment? I know there is jupyterhub and some people deployed it for selected trustworthy users (e.g. in a university course context or a company's intranet), but AFAIK it hasn't been deployed in production to potential untrustworthy users. Current approaches seem to rely on the idea to sandbox you in a Docker environment, but mostly these are work-in-progress solutions and I am not sure how much more secure they are, beside the fact that also authentication and persistent storage issues have to be solved.

The obvious problem is that accessing Jupyter is technically similar to allowing full shell-access and you have to deal with local privilege escalation, but I wonder if there has been any progress. I evaluated to use it as a UI for domain-specific applications that give users some kind of graphical shell, but in the end I decided against it, because of security concerns.

makmanalp• 9 years ago

try.jupyter.org is something like this, and I think it uses this: https://github.com/jupyter/tmpnb

takluyver• 9 years ago

You're right, it does use that. The security against untrusted users is whatever you get from docker - each user's code runs in an individual container. In the case of try.jupyter.org a breakout would not be catastrophic, though, because those servers shouldn't be holding any sensitive data.

carreau11• 9 years ago

http://mybinder.org/ does that, as well as https://notebooks.azure.com/ – well this one is authenticated, but it's easy to get a microsoft live account, and it does not even needs to be validated so that count (for me) as non trustworthy users. Technically giving access to Jupyter does not give full shell access. Only if the installed kernel give full shell access (which the default Python one does). The terminal feature can be deactivated as well.

jhoechtl• 9 years ago

Did you ever tried bootstraping an R kernel on mybinder?

dwrensha• 9 years ago

The IPython Sandstorm app is a step in this direction: https://apps.sandstorm.io/app/rprqf3t2h3vd3swfkhwk076qrennh9...

yuvipanda3• 9 years ago

You can use systemd or docker with JupyterHub, and make it as protected (or not) as you want. We could also write a spawner that spawns full fledged VMs, which would give you 'real' untrusted isolation - would that be something interesting to you?

ianhowson• 9 years ago

Here's one I made earlier: https://rnotebook.io

Unauthenticated, sandboxed notebooks in Docker containers. There are various limits.

If you manage to break it, please let me know!

(Regret incoming in 3... 2...)

carreau11• 9 years ago

You should get in contact with the mybinder.org crowd, they are low on manpower and are still looking for improvements.

williamstein• 9 years ago

https://cloud.sagemath.com deploys Jupyter into a multi-user environment. It's used by hundreds of courses for teaching across the world. Our Jupyter deployment also supports realtime synchronization (multiple people editing at once like Google docs) and recording of the complete history of the document. I've also spent the last few weeks on a complete rewrite of Jupyter from scratch using React to provide more robust realtime sync support, faster startup, and better integration with the rest of our platform; this rewrite is of course not live yet.

sandGorgon6• 9 years ago

This is incredible. I have been wondering what a react port of jupyter would look like. Is this opensource ?

Any learning from multi-user deployment ? We are trying to do this internally inside our company and jupyter hub is a little hard to grok.

I know of a lot of people who would pay for a faster jupyter that can also be run as a standalone dashboard/script - without the heavy duty interactive kernels. Basically reduce the prototype-deploy loop.

Check some of the comments here - https://news.ycombinator.com/item?id=14033129

zeptomu2• 9 years ago

I know sage math - in fact it was the first platform I found many years ago that provided an open alternative to Maple and Mathematica back in the days when Jupyter was still in its infancy.

It convinced me that there really exist viable open alternatives to the proprietary closed systems and I am extremely thankful for your efforts - I will definitely check out sagemathcloud.

lacksconfidence• 9 years ago

https://paws.wmflabs.org/paws/hub/login is available to anyone with a wikimedia account

int_19h• 9 years ago

accessing Jupyter is technically similar to allowing full shell-access and you have to deal with local privilege escalation

Some form of isolation - be it containers, or jails, or VMs - is going to be part of any solution precisely because of this.

I would actually dare say that FreeBSD jails are probably the best (most stable and secure) candidate of those available currently.

minimaxir4• 9 years ago

For those who use R, I strongly recommend looking into R Notebooks (http://rmarkdown.rstudio.com/r_notebooks.html), as there is a lot more versatility involved, especially over the Jupyter/IRKernel approach. Although it's R only, unfortunately (you can run Python code in it but not the way you expect)

I'd like to see some things ported into Jupyter from R Notebooks, like JavaScript data tables and the separation of code and output, making it easy to version control only the code. (Atleast this 5.0 release makes tables nonugly)

carreau11• 9 years ago

It would be great to sit with the Rstudio-notebooks developers and user and discuss how we could get the two platform to converge and be more interoperable. I guess both team have a lot on their plate, and we would need more manpower, but understanding and collaborating with other project for the good of user is always something the Jupyter team is happy to do.

imcoconut2• 9 years ago

The jupyter team has been working on jupyter lab which is a more r studio style environment. It incorperates notebooks, terminals, file editing, traditional ipython console, plus some other cool stuff in a web ide. You should be able to use an R jupyter kernel with it.

It's in alpha right now but they're making crazy fast progress on it. I've been using it and it's awesome

cercatrova2• 9 years ago

Something like Beaker? (www.beakernotebook.com)

karthik_ram• 9 years ago

I'm happy to connect you anytime! Just come by the west wing of the office. :)

robbles• 9 years ago

What does "a lot more versatility" mean? Just wondering what the advantages are.

Or do you just mean that R Notebooks are specifically better than the R support in Jupyter?

minimaxir4• 9 years ago

In addition to what I've already said, R Notebooks have native support for widgets, autogenerated TOC, and themes (although all of those can be added to Jupyter I believe). There's also the meta-support from being embedded in an IDE. (package management/variable explorer/etc)

Here's an example of one of my notebooks with all 3 things: http://minimaxir.com/notebooks/breach-network/

It's definitely a better fit for R.

peatmoss• 9 years ago

Well, for me the big one is that R Notebooks work cleanly with revision control systems like git.

That said, both Jupyter and R Markdown Notebooks are but a pale shadow of the support offered by Org-mode (seriously!).

jacquesm6• 9 years ago

Does it work in the same client-server fashion? (With Jupyter you can run the notebook on a server or the head machine of a cluster and then access the notebook remotely from your laptop or desktop.)

minimaxir4• 9 years ago

No. All local, although you could set up an Rstudio Server in theory and run the Notebook there, although I have less experience with that.

There are R packages which allow communicating with a server, mostly big data packages. (sparklyr allows you to connect to a remote Spark cluster)

dandermotj• 9 years ago

It's very easy to set up an RStudio server. I used to run one on Azure. Took minutes to set up and you get the whole RStudio suite, including R notebooks.

scottkuo• 9 years ago

How about both? https://notebooks.azure.com/

jimmcslim• 9 years ago

It would be great if Microsoft would open up a bit about what their plans are for this service, to the extent that it could be something I can rely on in a commercial context.

WeDontNeedH1Bs2• 9 years ago

Myself and all the Data Scientists I know use Python, while the Data Analysts in my company use R. Does R have anything like the Natural Language Tool Kit that Python has? Or SpaCy for fast NLP? Also, between PYKE, ProbLOG, and python-constraint, a lot of my logic programming needs are satisfied in Python. Does R have anything like that for logic programming?

minimaxir4• 9 years ago

Not as effectively. Which is why I use R for tabular data and Python for text/image/nontabular data.

And there's nothing wrong with knowing and using both languages.

rocho• 9 years ago

Just the other day I discovered that there exist a kernel (SOS kernel) that allows to use Python and R simultaneously in the same notebook (on a per-cell basis) and even has primitives for data exchange between the two.

I'm definitely using this for my next project.

a_bonobo• 9 years ago

You can also use R inside 'normal' Python notebooks via rpy2:

from rpy2 import robjects

my_dictionary_results = some_method()

names_dict = robjects.ListVector(my_dictionary_results)

%load_ext rpy2.ipython

%R library(some_lib)

%R -i my_dictionary_results doStuff(my_dictionary_results)

Image display etc. works fine

isolli• 9 years ago

In a similar vein: Beaker notebooks (http://beakernotebook.com/)

jacquesm6• 9 years ago

Jupyter is super useful, together with anaconda it's a winning team. I love anaconda because it takes the sting out of all the dependencies and cruft that stops python packages from installing cleanly (besides the v3 / v2 mess and installing as a regular user rather than root).

ciupicri2• 9 years ago

Also Anaconda tends to come with newer versions than those available in most Linux distributions.

superfx• 9 years ago

Still no collapsable hierarchy of cells. This is the one feature I miss most from Mathematica notebooks.

wannesm• 9 years ago

I can recommend http://jupyter-contrib-nbextensions.readthedocs.io/en/latest...

mFixman• 9 years ago

Also Table of Contents 2 [1] is useful, since it can numerate headings and subheadings in a natural way.

Jupyter extensions are incredibly powerful, and I don't know how I used to do data analysis before finding them.

[1] http://jupyter-contrib-nbextensions.readthedocs.io/en/latest...

sandGorgon6• 9 years ago

Does anyone know how to setup a multi user notebook? Because of the nature of the notebook and how it maintains state ... that a single instance is usable by a single person.

We are building dashboards in Jupiter and really would love it to be multi user... Without getting into the hub and stuff (way too complex to set it up)

yuvipanda3• 9 years ago

Disclaimer: I work on the hub.

Sorry to hear you found the hub too complex. We're working on making easier-to-use hub setups that fit different use cases. Can you tell us a little more about what your use case was and (optionally) which parts of the hub setup you found too complex?

Thanks!

sandGorgon6• 9 years ago

Thanks for replying.

So it's a bunch of different technologies - nodejs,etc. I'm kinda wondering if it can be built in Python itself. Make it part of a normal jupyter install, so just a "jupyter hub start " will work ?

EDIT: adding to that, you have built a nodejs based http proxy - can you not build it within Python (for uniformity) or nginx (for performance as well as mind share) ? Do you even need to mandate a http proxy ?

Second question is that can it run in a multiprocess - I don't want to run it in interactive mode, but just straight top to bottom. Perhaps there's huge memory savings there.

marcinkuzminski• 9 years ago

Looking forward to next OpenSource RhodeCode release which will now support rendering jupyter notebooks.

It'll be second to Github source code management with Jupyter support.

agumonkey• 9 years ago

The computational dom

m_mueller2• 9 years ago

I absolutely love Jupyter. Thank you guys and gals for your great efforts. I've had this idea about how it could become a very simple and powerful rapid application development tool based upon the widgets you already have: What if you could preview and bundle it in a mode with hidden code to standalone executable files on all platforms? There's already many tools in python to do this, all it would take is some export feature to tie it all together.

jasongrout• 9 years ago

We're going to be exploring building dashboarding solutions more with JupyterLab. See https://github.com/jupyterlab/jupyterlab/issues/1640 for some of our current discussion. That said, the deployment issue you've talked about has been experimented with in the IBM dashboarding features (i.e., one-click to deploy an application-like interface).

JupiterMoon• 9 years ago

Does this update break all the addons again? (Vim keybindings in edit mode, code folding etc.)

tmearnest• 9 years ago

It seemed fine for me. All of my addons were working.

wodenokoto• 9 years ago

I didn't understand the purpose of cell-tags. Can anyone give an example of how they might be used?

krastanov• 9 years ago

An example among others: I tag certain cells as "initialization cells" and they run automatically when a kernel is restarted (it requires a plugin).

krick• 9 years ago

Maybe it's just me, but the former table style seems much more readable and, well, compact, even if you claim otherwise. I hope, this behavior is customizable?

guidopallemans• 9 years ago

You can inject custom css through a ~/.jupyter/custom/custom.css file

wassyape• 9 years ago

I agree. It could be because our brains are used to the old style but there are some objective drawbacks.

The new style right aligns everything which is good for numbers but bad for text. Also, it might just be the screenshot, but the contrast is poorer and the column sizes aren't as fitted.

It's a pity, I would rather have a decent default than to customize every notebook.

jfdi• 9 years ago

If you want to install this as part of a more integrated sciences environment check out vnode https://github.com/thomaswilley/vnode

amenod• 9 years ago

One major issue for me is copying with mouse select + middle click, which is unfortunately not solved in this release. Ah well. :-/

Jupyter Notebook 5.0

Related Stories