smacke

I started burning down the backlog of all the stuff I wanted to get to for side projects but never had time for (before LLMs):

- https://smacke.net/ffsubsync -- automagically synchronize subtitles, now purely client-side in your browser thanks to pyodide

- https://ipyflow.github.io/ipyflow/lab/index.html?path=demo.i... -- reactive python jupyter notebooks, again in the browser thanks to pyodide / jupyterlite

- https://smacke.net/pipescript/lab/index.html?path=demo.ipynb -- magritter-like pipe / placeholder syntax for ipython / jupyter, again able to run purely in the browser

- https://smacke.net/pycograd/lab/index.html?path=pycograd_sim... -- pyccolo and pipescript-powered autograd, once again able to run purely in the browser since numpy has a wasm target (notice a theme here :) )

Some auto formatting italicized the text between * on different lines :')

Should be `reduce[$*$]`

Hi HN, I built pipescript over the holidays after I discovered magrittr for R and realized the same kind of pipe operator and placeholder syntax could be implemented for Python notebooks with a bit of creativity. I developed it while using it in parallel for Advent of Code and had a lot of fun with it; hope you like it.

Pickling + unpickling the object is a neat trick to update objects to point to the new methods, but it's even more straightforward to just patch `obj.__class__ = reloaded_module.NewClass`. This is what ipython's autoreload extension used to do (and still does in some circumstances, along with other tricks to patch up old references), though nowadays it's had some improvements over this approach: https://github.com/ipython/ipython/pull/14500

I developed a tool (https://github.com/smacke/ffsubsync) which can sync subtitles against each other (or even against an audio track), and this can be used in conjunction with other tools such as https://pypi.org/project/srt/ to combine multiple subtitle streams into a single stream. I've used this strategy to good effect to get both English and Chinese subtitles up at once.

Discussion on reddit from when I posted about this (more than 10 years ago now!): https://old.reddit.com/r/math/comments/144qc4/a_beautiful_pr...

Original pdf seems to have been taking down from AOPS but is still available in the internet archive: https://web.archive.org/web/20121018044426/https://artofprob...

You're probably referring to nbgather (https://github.com/microsoft/gather), which shipped with VSCode for a while.

nbgather used static slicing to get all the code necessary to reconstruct some cell. I actually worked with Andrew Head (original nbgather author) and Shreya Shankar to implement something similar in ipyflow (but with dynamic slicing and a not-as-nice interface): https://github.com/ipyflow/ipyflow?tab=readme-ov-file#state-...

I have no doubt something like this will make its way into marimo's roadmap at some point :)

ipyflow has a %%memoize magic which looks quite similar to %%xetmemo (just without specifying the inputs / outputs explicitly): https://github.com/ipyflow/ipyflow/?tab=readme-ov-file#memoi...

Would be cool if we could come up with a standard that works across notebooks / libraries!

I'm a big fan of Marimo (and of Akshay and Myles in particular); it's great to finally see a viable competitor to Jupyter as it can only mean good things for the ecosystem of scientific tooling as a whole.

I don't think anything here is in conflict with the statement that Fortran can be faster than C in some cases, but yes you're right that technically that should be qualified as naive Fortran can be faster than naive C

Fortran can be faster than C because it has a more restrictive memory model (compiler can make more optimizations when it knows nothing else is aliasing some memory)

<3

Another giveaway is the ratio of stars to watchers / forks. I remember one project with thousands of stars but only 10 users "watching" it. They went on to raise a sizable seed round too.

Not necessarily indicative of foul play. I have two projects like this (https://github.com/smacke/ffsubsync and https://github.com/ipyflow/ipyflow) and I attribute it to not having great developer documentation.

Is the recursion limit an implementation detail of the compiler though, or something statically built into the language? If the former there's nothing to prevent the language itself from being Turing complete.

(Similar to how every Turing complete programming language today runs on something technically weaker than a Turing machine due to finite memory.)

Ah yeah, Carlos is very low on cycles (like all of us I guess), probably due to the day job. (If only OSS had competitive pay...)

Have you opened an issue against https://github.com/python-lsp/python-lsp-server? I'm an occasional contributor for easy fixes; if your fix is easy chances are it would get picked up.

It's not in the highlights, but one of the things that excites me most is this: https://docs.python.org/dev/whatsnew/3.12.html#pep-669-low-i...

PEP 669 defines a new API for profilers, debuggers, and other tools to monitor events in CPython. It covers a wide range of events, including calls, returns, lines, exceptions, jumps, and more. This means that you only pay for what you use, providing support for near-zero overhead debuggers and coverage tools. See sys.monitoring for details.

Low-overhead instrumentation opens up a whole bunch of interesting interactive use cases (i.e. Jupyter etc.), and as the author of one library that relies heavily on instrumentation (https://github.com/ipyflow/ipyflow), I'm very keen to explore the possibilities here.

Didn't try downloading the SDK yet, but in the playground, seems like the "Python superset" advertisement does not hold (trying to define a class gives "error: classes not implemented yet"). Great to see progress in this space but I think the real killer feature of Mojo would be to run any existing Python program and allow programmers to incrementally adopt Mojo's syntax for targeted optimizations, and seems like we're not there yet.

What would a "composable" experience look like?

Err, I guess Pluto probably needs the round trip too but I guess Julia is faster after the initial jit...

TY for the feedback; updated the readme to try to get the value proposition across more quickly.

I would be very surprised if something like this gets support in core Jupyter -- there's a lot of added complexity. Fortunately it is doable as extensions for Jupyter / JupyterLab.

We have a couple of papers that go into some of the details.

https://smacke.net/papers/nbsafety.pdf https://smacke.net/papers/nbslicer.pdf

It looks like you are using a static approach for dependency inference. There are a lot of benefits to static approaches, but they can only get you so far. My JupyterCon presentation includes a bunch of examples where dynamic approaches are a must: https://t.ly/78rS

Besides that, there are a bunch of interesting design decisions about when to add edges between cells, when to break them, what metadata to annotate edges with, etc.

I'm hoping to abstract a way a bunch of the complexity by developing something like a runtime version of a language server protocol (working name "language kernel protocol") so that any editor that implements the protocol would get reactivity for free when running a kernel that likewise implements the protocol. I have an early version of this which is how IPyflow works for both Jupyter and JupyterLab; VSCode would be a great editor to add support for next.

Personally I don't like the "write to disk" approach; I think it kind of just punts the state problem somewhere else (i.e. from memory to disk). Writing to a database and adding versioning is better, but that's a lot of machinery to expect a notebook user to adopt (though maybe better tooling could help). Also a lot of Python objects are not out-of-the-box pickleable (e.g. generators). Also pickle is a mess.

"solved" is a very generous adjective

Thank you! If you had tried to use it before, it probably would have broken pretty quickly. Now it will still break, but not so quickly as to not be useful, hopefully.

External contributions are mostly blocked on me right now to improve both user and developer docs (improve = write the first draft in this case).

For this project, Python is a hard requirement, though it's possible the approach may be applied (after significant effort -- Python took me ~3 years and counting) to other languages / runtimes as well.

IPyflow takes a round trip from client to kernel for each execution (including reactive executions) -- this approach is necessary to get the best possible accuracy when determining dataflow in a highly dynamic language like Python, but it is an architectural limitation that prevents the reactivity from feeling as instant as in Observable or Pluto.

It's very new, and the current frontend implementations for Jupyter and JupyterLab include some workarounds for fundamental protocol-level limitations that probably make adding this kind of feature as part of the core package a no-go (without first addressing the core protocol limitations).

Yep that was one of the talks that motivated this work -- talked about it in my JupyterCon talk today :)

https://t.ly/78rS