Hotcaml: An OCaml interpreter with watching and reloading

by todsacerdoti • 5 years ago

122 31 5 years ago

dwohnitmok5• 5 years ago

The thing I'm always curious with hot-reload systems is whether there's a good story for state change. What happens if a `ref`'s type is changed? Are `ref` values preserved between reloads?

EDIT: Ah I just read https://github.com/let-def/hotcaml#synchronous-and-asynchron... so it looks like at least for the synchronous `hotcaml` bit it just completely reloads from scratch so state is not preserved, but I'm curious then how the `hotcaml_lwt` frontend deals with this issue.

rsstack• 5 years ago

Check out how Erlang/Elixir/BEAM deal with this. Stateful processes write a state-migration function that's run (new code, old state) when hot reloading the process. Requires some forethought, especially when processes can be updated in any order, but it's really amazing. Of course, when doing hot-reloading in a local development environment and not in production, it doesn't make as much sense.

dwohnitmok5• 5 years ago

Right Erlang has explicit migrations and dynamically-typed image-based languages just persist the state.

I'm curious if there have been any similar things for statically-typed languages.

munificent2• 5 years ago

The Dart VM supports stateful hot reload and it's used heavily by Flutter developers. There are some kinds of structural changes that necessitate a full restart that wipes state away, but the VM is suprisingly good at migrating the existing objects in the heap to match any changes to the underlying classes and preserve as much state as possible.

https://docs.flutter.dev/development/tools/hot-reload

sitkack2• 5 years ago

I was going to make a similar comment about reload() in Python. One dev technique out of the many available is to have your code being edited in your editor while in another terminal window, using an ipython repl to drive it. Currently one would need to `from importlib import reload` and of course ipython itself has

https://ipython.readthedocs.io/en/stable/config/extensions/a...

Launch ipython like you would `pdb`

https://pypi.org/project/ipdb/

*edit to @dwohnitmok as the author of Crafting Interpreters mentioned, it really works pretty well in practice. And for cases where I was changing enough state that it bails out, you can either set a breakpoint in a test or create a little closure to build up the right state and then pass back an fn to step through in ipython.

Infact, this dev style reinforces an immutable nearly functional data centric programming model. With that model, changes to code won't impact the data, so reloads are actually rare. Have shallow reference chains, use value types in simple containers and lots of problems kinda melt away.

Ha! https://news.ycombinator.com/item?id=29965226 the highest rank post on Google search for that quote is less than 3 days old. Know we know why results are so bad!

dwohnitmok5• 5 years ago

Right so I think this the approach that most UI frameworks tend to adopt, which is to have a central data store that then is persisted between changes to the view code that generates a view from that store (this is effectively how hot reloading works for things like Redux with Webpack or Elm and I think based on those docs, also how Dart does it).

I'm curious though what happens if you have to make changes to that data store in a statically typed language like Dart, e.g. change the type of a field. The docs there only seem to have a few sentences about that (which I think is talking about a related, but slightly different thing, which is that hot reload allows a whole new host of paths to create state which might've been impossible to ever achieve during the normal, non-hot-reloading operation of an app, including violating certain assumed invariants):

If code changes affect the state of your app (or its dependencies), the data your app has to work with might not be fully consistent with the data it would have if it executed from scratch. The result might be different behavior after hot reload versus a hot restart.

Does the app crash? Or does it automatically restart with a fresh slate (maybe this falls under the "some kinds of structural changes that necessitate a full restart that wipes state away" you mentioned)? Or is the change ignored? Or is there some degraded "delayed type checking" mode that occurs with hot reload that changes a type into a runtime type guard and causes some sort of exception for anything that requires the field to be a certain type?

def-lkb3• 5 years ago

Both the synchronous and asynchronous frontends preserve as much state as possible (all untouched modules are kept). They differ in the control flow they permit (the asynchronous one being more expressive).

I will try to make a parallel with a simple Makefile-driven project.

In the initial build, all files are built, following a topological ordering of the dependencies. For the Makefile it means invoking some unix command for each intermediate artefact. For Hotcaml, it means parsing, typing, compiling and loading each module.

After that, build is incremental: when a file is modified, only this file and its reverse dependencies need to be rebuilt. Make will keep the existing artifacts that are not out of date. Hotcaml will keep the existing modules that are not out of date. And their state is preserved.

Now, the synchronous frontend is quite like a regular Makefile: a build step is considered done when the build command finishes. The asynchronous frontend allows command to run in the background, like a Makefile that would spawn processes/daemon using the shell & control operator.

In Hotcaml, these threads continue executing concurrently with the reloading process. They can be notified that their "host" module has been reloaded by registering with some runtime service (via a module named Hotlink).

scoopr• 5 years ago

Also Visual Studio does `Edit and Continue`[0], which I believe works by patching function prologues to jump to another location.. I don't think it handles structure changes all that gracefully, but honestly I haven't used it that much to know for sure.

[0] https://docs.microsoft.com/en-us/visualstudio/debugger/edit-...

LAC-Tech• 5 years ago

awesome.

It's been a real PITA to crash out of an interpreter then reload it again. Dune helps in that it will load the libs for you, but you've still got to open modules etc.

rixed• 5 years ago

You can write an init file to perform all the required loading and configuration if that happens often.

Or create a custom top level if for some reasons you prefer the extra modules to not be distinct files.

malkia• 5 years ago

btw, there is an awesome podcast on ocaml - https://signalsandthreads.com/

maattdd• 5 years ago

It's not specifically on OCaml.

brightball• 5 years ago

Is this pronounced “hot-ca-mal-e”? If it’s not it really needs to be.

cglong• 5 years ago

Or renamed to "Sahara"!

synaesthesisx• 5 years ago

OCaml is fascinating - it was my first exposure to functional programming through a great class at Harvard. We had a guest lecture from someone at Jane Street that was quite memorable. Although these days I'm uncertain of any large institutions (outside of Jane Street) that actually use OCaml.

FluffyFez• 5 years ago

I really do wonder why Ocaml hasn't had its rails moment.

I hated ML when I first encountered SML in college. As I became more familiar with functional programming, I started enjoying them. the ML languages just felt like such a better deal than the complexities arising from the purity of Haskell.

More intuitive and have records. Maybe they'll have a moment yet. Their learning curve might still be steep. I do hope for a wide spread variation to come out some day.

lmm• 5 years ago

Well, it hasn't had anyone implementing something like Rails - it's easy to underestimate the importance of making a really simple onramp for beginners. I think there's a lot to be said for gradual sustainable growth though; OCaml isn't hyped, but we're quietly seeing more and more of it.

mirekrusin• 5 years ago

Michael Ryan Clarkson's from Cornell (CS 3110 Data Structures and Functional Programming) [0] is publicly available, really great, short videos format.

[0] https://www.youtube.com/playlist?list=PLre5AT9JnKShBOPeuiD9b...

ssfrr• 5 years ago

I’ve been learning OCaml over the past week or two and these videos have been extremely helpful. They’re well-organized well-edited. Seems like a ton of work went into them.

chrischen2• 5 years ago

All the companies using Rescript (formerly ReasonML + Bucklescript compiler) and ones using Reasonml right now are effectively using Ocaml. It allows you to run Ocaml but target web browsers or Nodejs.

This includes apps like TinyMCE, ahrefs, and around 50% of messenger.com code [1].

[1] https://reasonml.github.io/blog/2017/09/08/messenger-50-reas...

yawaramin2• 5 years ago

Just fyi, Ahrefs and probably some others in that list were using OCaml way before Reason came along. Ahrefs' web crawler (the second biggest after Google) is actually written in OCaml (native).

junon• 5 years ago

Most of the verification community relies on OCaml to some extent.

nextaccountic• 4 years ago

Funnily enough, there is no verified OCaml implementation yet. Unlike, for instance, SML (well CakeML is a subset of SML with a verified implementation)

Hotcaml: An OCaml interpreter with watching and reloading

Related Stories