Interactive workflows for C++ with Jupyter

https://blog.jupyter.org/interactive-workflows-for-c-with-jupyter-fe9b54227d92

by SylvainCorlay • 9 years ago

292 90 9 years ago

blog.jupyter.org

ddavis2• 9 years ago

Very cool! I watched the SciPy talk on xtensor (same developers as xeus) and the project looked pretty awesome. Many people in my field used CINT for a long time and cling is so much better. It's nice to see it used outside of particle physics.

One thing I disagree with is the idea that the lack of a good interactive environment for C++ makes it difficult to teach. C++ is a compiled language, so learning how to compile a C++ library or program is _part_ of C++. I feel like C++ beginners should go interactive _after_ learning how to compile a project. I write that with first hand experience; my first programming experience was with writing interpreted C++ in CINT and I feel like it hindered my ability to eventually understand what a real C++ program was.

pjmlp7• 9 years ago

C++ on environments like C++ Builder is relatively easy.

The issue is developers having open mind to such environments, luckily it seems to be changing with QtCreator, CLion, VC++, specially thanks to clang being implemented as a library.

jmabille• 9 years ago

I agree that the compilation process is part of the C++, however (and that's from my teacher's experience), teaching it to beginners as a first lesson may be complicated (and you have to teach it early so they can build their first program). Besides this is not rewarding when your students are familiar with other languages like python where they can write some code and have immediate results without any additional step.

So having a good interactive environment is really useful for the first lessons, then you can teach the compilation chaintool and switch to a real environment.

xfer• 9 years ago

Compiling a project with right incantation of flags is not rocket science but it is time consuming that a build system should be better equipped to do.

Interactive development has immense benefits in exploring apis for experts and learning the language for beginners. And C++ is not well-known for its compile-times.

jcelerier• 9 years ago

so learning how to compile a C++ library or program is _part_ of C++.

I entirely disagree. This is only for historical reasons, the language itself does not care at all about the compilation model. This mindset is what is keeping C++ back, both for the industry AND in students' minds.

I feel like it hindered my ability to eventually understand what a real C++ program was.

The moment you have a .o file it's not a C++ program anymore, but a platform-specific object file. You aren't learning C++ but windows / mac / linux's native binary production toolchain.

AstralStorm3• 9 years ago

Indeed, the language does not care which means you should. Object file is not even a compilation unit at times.

You get to build the object file, shared objects, dlls what have you. Then use the linker to link it to turn it into an executable.

Some modern languages specify the runtime very deep. For example Python or Java - so much that it is hard to separate language from runtime or standard library.

ddavis2• 9 years ago

The moment you have a .o file it's not a C++ program anymore, but a platform-specific object file. You aren't learning C++ but windows / mac / linux's native binary production toolchain.

Then that is at least part of the learning curve of seriously using C++ so I think as of today it's (potentially) dangerous to start with only interpreted code and wait to be introduced to compiling, build tools, etc. They should be taught together. I also think it's not the greatest environment - and I hope it changes, perhaps projects like this will help!

dguest• 9 years ago

Someone else commented on the semantic difference between the narrow definition of C++ as a specification and the looser "C++" as a part of the full stack.

With regard to the full stack, I tend to agree with you, at least in the academic environment. Organizations that don't have a lot of software engineers will shoot themselves in the foot if they focus on the C++ language rather than the infrastructure around it. An interactive notebook is great to teach someone to add a few lines of code here or there, or to run quick tests, but what we're lacking is people who understand how to plug their jupyter notebook into the production line.

FlyingSnake• 9 years ago

C++, despite being a really powerful language, has a really primitive build system. Developers have to wade through a complex soup of Makefiles/CMake/Gyp/ninja etc just to get a project set up. There's conan for package management, but it pales when compared to Cargo or Gradle. Moreover setting up Conan is a full time job in itself, and it's not without weird quirks either. I've not tried Bazel yet, and would love to hear if it's as good as Xooglers say.

C++ on Jupyter looks amazing, and I really hope it makes starting with C++ easier.

coherentpony2• 9 years ago

Developers have to wade through a complex soup of Makefiles/CMake/Gyp/ninja etc

But you do this precisely once.

mschuetz2• 9 years ago

Once every week.

thenewwazoo• 9 years ago

I'm a bit surprised to read this. It's been a long time since I've done C++ outside of the embedded space, but isn't this a problem that's solved by libraries and LTO? Compiling individual source units isn't terribly difficult either. There's a single tool that turns source files into object files, and a single tool to link them all up. It all feels very UNIX-y to me.

weberc24• 9 years ago

LTO is unrelated. Compiling individual units is only one of many build system responsibilities. There's also dependency management, cross-platform support, iterative builds, etc. And it all needs to be fast.

harry82• 9 years ago

Cake https://github.com/dcasnowdon/Cake

Written by Matt Hermann who I believe is now at google - I've never met him. I worked somewhere that used it and have used it ever since for every greenfields C++ project I've done and just don't think about my build system anymore. For me this just works for everything I need, adding a linkflag for this .cpp file only? Dead simple, hard to imagine it being simpler. Build tests and run with valgrind equally so. I have a single top level makefile that literally just specifies targets, eg release, test, asan_test, clang_release etc. Each is a single cake line.

I highly recommend it for C++ on linux (haven't used it elsewhere to say), for me it's a thing of beauty and a joy forever. Not dealing with autotools or scons or complicated make based build systems makes C++ so much nicer to work with.

malkia3• 9 years ago

Bazel is what I really like (xoogler here, used it mainly with java+protos, minor python/c++, testing, other things). Similarly - gn (though different internals, easier platform switch, etc.), buck, pants, etc.

In all of these I like the single "BUILD" file per directory, and simple text file explaining dependencies.

Also these tools work awesome when you are in a company, and you have underneath - dozen or more libs, sdks, projects that need to run together.

Matthias2474• 9 years ago

Also these tools work awesome when you are in a company, and you have underneath - dozen or more libs, sdks, projects that need to run together.

I'm pretty sure it works great then. But for the bigger C/C++ ecosystem the problem is that everyone uses another build system. From Makefiles to Automake, Cmake, Gyp, Scons, Bazel, etc. everything is included. As soon as you try to use dependencies that favor a different format you either have to rewrite their project definition in your format or at least build it as an external subproject and manually add the include directories.

Imho that's one of the biggest weaknesses of C++ that even outweighs language concern for me: It's often lots of work to integrate 3rd party libraries. And because of this there isn't a really great ecosystem of libraries (e.g. compared to Javascript and Java).

curiousgal• 9 years ago

Gradle

Ugh

superbatfish5• 9 years ago

The conda package manager and it's subproject conda-build are fantastic tools to solve this problem. I write scientific software (primarily a mix of C++ and Python), and I use conda for literally every dependency I need, including pure C or C++ dependencies. Most of them are already available via the conda-forge project, and the ones that aren't are easily integrated by writing my own recipes to package with conda-build.

dimatura2• 9 years ago

I write in a mix of python and C++ as well. As such, I've also gradually moved towards using conda. What's your dev workflow for the code that's not a dependency like? Specially if the code you're developing is C++. Do you make a conda package of it as you're developing? I've found that too much of a hassle to rebuild. I usually just use CMake with the conda environment dir as the CMAKE_INSTALL_PREFIX.

ryanpepper2• 9 years ago

I've found generally it's fine for most things but issues come from when the scientific libraries are built with/without certain options. The lack of OpenMP for FFTW is annoying, for example.

boris3• 9 years ago

Check build2, it has a sane build model and syntax and comes with integrated package manager/repository, similat to Cargo:

https://build2.org

rtpg2• 9 years ago

Yeah I'm a bit sad about build systems.

I wanted to write a Postgres extension/fork, and was thinking "oh I'll just replace some bits with Rust".

Unfortunately everything is `make`, which is great when you're setting something up and have a huge process... but it's very hard to make some changes without figuring out the entire flow.

Someone that could somehow wrap make to make it easier to make incremental changes on existing projects could help fuel a lot more experimental changes.

boris3• 9 years ago

... wrap make to make it easier to make ...

I think there is a deep truth in (unintended?) pun: those who don't understand make are bound to reinvent it, poorly.

keerthiko• 9 years ago

C++ was the first programming language I picked up that made me feel like I was really making something (after VisualBASIC and HTML), when I was 13. Things were simpler back then. I used Borland C++ and it could produce .exe exports from .cpp sources, pulling in referenced #includes. Running it in Windows would launch a little DOS prompt that ran your program and then quit (unless you put a getch()!)

Like any teenager tinkering, I opened up some sample cpp file that came with the compiler and started just throwing in cout<< all over the place doing random stuff with variables. My older brother explained "loops", "functions" and later the basics of "arrays" and I completed my first game by 16, written purely in C++ -- a Tank clone that was a single long-ass cpp file using purely #include<graphics.h> calls, putting individual pixels on a 640x480 screen. I hadn't learned what classes were yet, so it was some form of "functional programming", just starting from a void main()

The point was, it was easy to tinker and learn and make stuff and explore in C++, the language isn't innately arcane. I didn't even have the internet back then to look stuff up. A teacher gave me a textbook for "graphics in C++" because my high school (which I wasn't in yet) computer science curriculum didn't cover graphics, but I hated copying code from the textbook (it seemed outdated) so I just opened up header files and tried to call functions I saw in there. I didn't need to use any fancy C++ syntax, just the basics my brother taught me. No make files, no build scripts, no gcc. Just an IDE that existed on a floppy disk and the most common OS available at the time (DOS).

It must be a sign I'm getting crotchety and old, but "back in my day" C++ wasn't such a confounding beast. And I'm sad to say I think modern-day python is going down a similar path, it started out intending to be the easiest language to tinker in, but it's slowly becoming arcane just to import a module and launch a program for a beginner.

I stepped back into C++ recently trying to make something "I can see" at from scratch, and it's damn nigh impossible without visiting 4-8 different instructional sites depending on how you define "something I can look at".

pjmlp7• 9 years ago

Picking up your example, that is exactly why ANSI C++ people want to have some kind of basic 2D support on the standard.

Yet the naysayers state that apparently we can have standard IO, filesystem and networking support, but not 2D.

The reason being that not all scenarios have graphical displays, as if all deployment scenarios always have filesystem and networking stacks.

cygx• 9 years ago

it was some form of "functional programming", just starting from a void main()

'Procedural' is the term you were looking for.

periram• 9 years ago

Finally! Thanks a ton to both Cling developers and Jupyter. Personally, a lot my programming involves prototyping and playing with a lot of different ideas and implementations before settling on a solution. Compiled languages put a steep barrier to this way of thinking and that is why Python has always been my favorite. I have come back to C++ thanks to Cpp14, 17 and great new features like this!!

rocqua3• 9 years ago

This looks downright amazing.

I only wish it had come a few months earlier. I've had to switch from python in a jupyter notebook to C++ for performance reasons, and I really felt the loss of incrementalism.

cbcoutinho2• 9 years ago

You can create your project into python library by wrapping in ctypes or using the cffi package. I do that with some of my numerical simulation code and it doesn't take away from the interactivity of python. Once I get a feature more or less complete, I re-write it in c/fortran and compile it away. This maintains performance as well as focus, because the 'interesting' thing you'll be working on is quick and dirty using python, and the stuff you've already okay'd is in a compiled language out of sight.

rocqua3• 9 years ago

It is this transpiling from python to C++ that I want to avoid. Especially because there is a lot of iterative development on the C++.

However, I'll essentially never be able to run my code from a notebook because it runs on a cluster I need to SSH into. Thing is, I'd like to develop that code locally and iteratively.

BeetleB• 9 years ago

It would be great if we get C++ on Jupyter. At the moment, I use Org Babel to do these kinds of things. It's a halfway solution.

mcemilg• 9 years ago

Okay. If its works amazing, so it can be work on other compiled languages? This will be amazing and it can be game changer.

kasperset• 9 years ago

How is it different from R notebook implementation of C++ chunks execution?

peheje2• 9 years ago

Cool. For anyone looking at using Python when speed is important should also take a look at Numba.

superbatfish5• 9 years ago

Yes, numba is great; I've had a good experience with it for some functions.

Some problems aren't suited well for it though, especially those that involve Python dictionaries or sets. For those problems, switching to C++ results in MUCH faster code (std::unordered_map is way faster than Python's dict, especially for small elements).

peheje2• 9 years ago

As I'm aware of Python dictionaries are not supported by Numba, however it will still run in "object mode" which won't give you much performance boost. I tend to always specify the Numba function as njit or objectMode=False. But yea true, Numba does have shortcomings, maybe dictionaries and classes will be supported in the future.

muxator• 9 years ago

Most of the progress made in software projects comes from incrementalism. Obstacles to fast iteration hinder progress.

I would like to print it in very big letters and put it at the entrance of slow-moving IT departments. "cost of change" is likely one of the most important metrics to measure the effectiveness of a team.

Being risk adverse because of fear of change leads to stagnation.

"more Jupyter notebooks and less spreadsheets" could be a useful simplification for conveying this change of POV.

taeric3• 9 years ago

And yet,"more data in spreadsheets" is a truism. The spreadsheet model is empirically one of the most successful ones we have.

Worse, so called "notebooks" are neat, but they are just as bad for engineering best practices as spreadsheets are.

Not to say that you shouldn't be able to grow from freeform practices. You should. Just don't mistake them for being somehow superior to spreadsheets.

jononor2• 9 years ago

You can very easily move some functions out of a notebook and into a proper module, that can be included in a program / production system. Autoreload extension in Python Jupyter notebooks makes it extra nice. It is also easy, and convenient to write py.test style tests inside a notebook, just call them manually inside the block. Then when you move it out to module, you have tests that will automatically be picked up by your testrunner.

aldanor• 9 years ago

I can’t help mentioning my own project - https://github.com/aldanor/ipybind - which provides a lightweight Jupyter interface to pybind11 [1], allowing to quickly sketch working C++ extensions in the notebook; all features of pybind11 like numpy support are immediately available. Been using it myself quite a lot for prototyping C++/Python code.

[1] https://github.com/pybind/pybind11

harry82• 9 years ago

This is how I'm doing C++ rapid interactive. Combined shell and C++ in a single file that compiles itself and either runs itself or runs objdump on itself. I find it really useful and fast to test something in junkcode. Not quite the same problem as interactive C++ with Jupyter but there's some overlap. Overlap with godbolt as well - find this quicker perhaps because I'm more used to it?

https://github.com/hal88/junkcode/blob/master/template_scrip...

I post in the hope that it is a useful idea to someone and also that people respond with their similar ideas that might be useful to me.

rb808• 9 years ago

I'd like to use Jupyter notebooks to do support tasks - eg check on system hosts and do reports, maybe restart processes. I think it could be perfect. I'm just hesitant because no one else seems to use it this way. Any advice out there?

gcc_programmer• 9 years ago

I think jupyter with C++ is an amazing idea, and I hope it will help more people to start with, in my subjective opinion, one of the most important programming languages today.

That being said, I look down on all the commentators who say that C++ has a primitive build system/package manager etc. Lets be clear: C++ has no package manager, and , in my view of the world, no compiled language has a native build system, its just that most compiled languages dont separate compilation from linking and also most languages support modules :)

C++ is hard. Throwing some fancy frontend and and a REPL will only make you realise that sooner. The complexity is mostly unjustified, but the language is so powerful that you could do anything with it: from trading strategies to power plant control systems to game engines. I have been studying C++ for 9 years, have used it for 7, and have been a professional developer for only 2.5 years, and I think I have only scratched the surface. I have used python for a lot less time but I feel like I am more comfortable with it because Python is trivial to learn (to a person with a comp sci background ofc). What you think you are cutting away (dependencies/build jnfo) is something essential to learning C++ and how everything comes together. Better learn it sooner than later imho.

Rant over

weberc24• 9 years ago

no compiled language has a native build system, its just that most compiled languages dont separate compilation from linking and also most languages support modules :)

Most modern, mainstream compiled languages have a build system that wraps around the compiler and linker. Invoking the compiler and linker are responsibilities of the build system, and frequently modules are compilation units.

Modern C++ is a neat language (I used to be a C++ developer) but it's tooling alone makes it unsuitable for any project where rapid iteration is remotely important.

blackflame70003• 9 years ago

You're right about most compilers having a build system MSVC being probably the biggest.

Modern C++ when combined with Qt makes rapid iteration very doable. In my opinion Qt should be the standard c++ model.

mempko• 9 years ago

Looks like the world is finally catching up to the Future that Temple OS lives in.

Mathnerd314• 9 years ago

TempleOS doesn't do scatter plots.

westurner• 9 years ago

QuantStack/xeus-cling https://github.com/QuantStack/xeus-cling

QuantStack/xwidgets https://github.com/QuantStack/xwidgets

QuantStack/xplot (bqplot) https://github.com/QuantStack/xplot

boltzmannbrain• 9 years ago

I really hope this works as promised. I'm curious if they'll add support for boost (www.boost.org) though.

deepGem• 9 years ago

Actually, I've found Xcode to be a very easily usable prototyping tool for C++. I haven't used it in any large project, but it's super simple to just prototype some concepts. Having that in a browser is even better.

TisenoT• 9 years ago

This also makes C++ more difficult to teach. The first hours of a C++ class are rarely rewarding as the students must learn how to set up a small project before writing any code. And then, a lot more time is required before their work can result in any visual outcome.

Install a c++ compiler, create main.cpp, `g++ main.cpp`

What am I missing?

flukus• 9 years ago

What am I missing?

The students don't know anything about the CLI. They might not know what a directory is or how to cd to it. They may have to install multiple packages. On certain operating systems they'll have to set some PATH variables.

I'm not saying it shouldn't be that easy, but due to failures in earlier education it's not.

ryanpepper2• 9 years ago

When I was taught C++ at University, we were just told to download Quincy which bundles a (pretty old) version of GCC with a text editor on Windows.

Once we had the basics down they introduced Linux, the command line interface to GCC, etc.

scardine• 9 years ago

I could not make it work on a mac. It almost works but looks like there is no mac package for xeus-cling at the conda repos.

Tried also to clone the github repo and compile locally but got tired of fixing dependencies and stuff after a few compiler errors.

rawland• 9 years ago

To install this, make sure to start with plain miniconda2, instead of anaconda2.

I was fighting against QuantStack's modified software stack for two hours until I discovered this:

https://github.com/QuantStack/xeus-cling/issues/74#issuecomm...

Fortunately, they updated the README.md in

https://github.com/QuantStack/xeus-cling

superbatfish5• 9 years ago

Good tip! IMHO, I always recommend the Miniconda installer over the full Anaconda installer, except for absolute beginners who would struggle to understand basic concepts like environments or the PATH environment variable, etc.

Interactive workflows for C++ with Jupyter

Related Stories