talex5

https://www.youtube.com/watch?v=Hw-_x9CfqA8&list=PLyrlk8Xayl... - starts around -4:30:00 (but I guess that's relative to now and will change).

I just tried it. :X asks you to enter an encryption key, then asks you to enter it again, and only continues if the keys match. And then you're still in vim and need to save your file to overwrite anything. Seems hard to do by mistake.

(though I prefer ZZ)

Here's an example using OCaml 5 to run multiple fibers concurrently (look - no monads!):

https://github.com/ocaml-multicore/eio#fibers

I am not sure ARMv1 (and v2) even had supervisor vs user mode, etc.? (It may have, Google isn't helping me here)

v2 at least had 4 modes: user mode, supervisor, IRQ and FIQ. They were encoded in the low 2 bits of r15 (which weren't otherwise needed, since the PC was always word-aligned).

The original Archimedes was intended to run a "preemptively multi tasking, multi-threaded, multi-user, written in Acorn-extended Modula2+" system called ARX. But when that wasn't ready in time, they ended up just porting the 8-bit BBC Micro MOS to the new 32-bit machines!

This is a great read about it all:

http://www.rougol.jellybaby.net/meetings/2012/PaulFellows/in...

Work has already started: https://github.com/ocaml-multicore/eio

There's also now https://github.com/talex5/lwt_eio, which allows you to run existing Lwt code alongside code using effects, to aid with porting.

Instead of Stdlib.Queue you can use Eio.Stream, which is thread-safe (and will take care of waking sleeping threads when data becomes available).

The README shows an example of a pool of workers pulling jobs from an Eio.Stream:

https://github.com/ocaml-multicore/eio#example-a-worker-pool

We're still exploring what APIs to provide for this kind of thing, and in particular how to unify domainslib and eio.

I usually notice if my blog gets on Hacker News fairly soon, so I'll see any comments posted here.

I prefer not to use email for most things because then the reply only benefits one person, whereas replies in a public forum can be read by others and get indexed by search engines.

I've been thinking about creating a Matrix room for each blog post as a discussion forum, but so far most posts have ended up on other discussion sites anyway.

To clarify that, there are two systems here:

- domainslib schedules all tasks across all cores (like Go).

- eio keeps tasks on the same core (and you can use a shared job queue to distribute work between cores if you want).

Eio can certainly do async IO on multiple cores.

Moving tasks freely between cores has some major downsides - for example every time a Go program wants to access a shared value, it needs to take a mutex (and be careful to avoid deadlocks). Such races can be very hard to debug.

I suspect that the extra reliability is often worth the cost of sometimes having unbalanced workloads between cores. We're still investigating how big this effect is. When I worked at Docker, we spent a lot of time dealing with races in the Go code, even though almost nothing in Docker is CPU intensive!

For a group of tasks on a single core, you can be sure that e.g. incrementing a counter or scanning an array is an atomic operation. Only blocking operations (such as reading from a file or socket) can allow something else to run. And eio schedules tasks deterministically, so if you test with (deterministic) mocks then the trace of your tests is deterministic too. Eio's own unit-tests are mostly expect-based tests where the test just checks that the trace output matches the recorded trace, for example.

The Eio README has more information, plus a getting-started guide: https://github.com/ocaml-multicore/eio/blob/main/README.md

I tried a couple of different AMD cards, and my machine crashes on resume if I try to use either of them (but the Intel iGPU works fine).

Searching for amdgpu bug reports leads to:

https://amdgpu-install.readthedocs.io/en/latest/install-bugr...

which links to a page saying "Bugzilla is no longer in use" :-(

This is under Qubes/Xen, though, so maybe that causes extra problems. If any devs are reading, I did report it here in the end:

https://github.com/QubesOS/qubes-issues/issues/5459

By default, opam installs everything into the current "switch". Typically you have switch one per compiler, but you can create one per project. You can also create a "local" switch directly inside your project directory: https://opam.ocaml.org/blog/opam-local-switches/

Another option is to use the duniverse tool (https://github.com/ocamllabs/duniverse). That downloads all dependencies to the project directory, and then builds them all together as one giant project. That makes it very easy to make changes across multiple libraries at once, while keeping fast incremental builds.

And a final option is to build projects with Docker, which allows you to have separate versions of non-OCaml packages too.

Indeed. The project has since moved under the mirage org on GitHub, and now has several contributors:

https://github.com/mirage/qubes-mirage-firewall

For reference, the ugly OCaml equivalent unikernel is:

let () = print_endline "Hello, world!"

(from http://roscidus.com/blog/blog/2014/07/28/my-first-unikernel/)

the recommended tools never seem to work together on anything but the latest version

Whenever a package is added to the repository, the CI tests (the latest compatible version of) every package that depends on it to check everything still works. However, non-current packages may break if they're missing upper bounds (it doesn't check all previous releases).

However, since opam-repository is just a Git repo, you can clone a known snapshot of it at some point in time and keep using that. That allows you to continue using old versions of packages without the risk of new software causing unwanted upgrades.

My impression is that INRIA is too eager to add features and tweak the language.

That's strange. I've always had the opposite impression - that they maintain compatibility at all costs, including leaving sub-optimal APIs in the standard library.

Next, everything builds so slowly, and just trying to set up Core, utop, and ocp_indent is a trial in patience, watching the same dep build again and again

The Core alternative standard library is pretty huge, indeed (I don't use it). I just tried installing all the tools you mention in a fresh container:

$ time docker run ocaml/opam:debian-9_ocaml-4.05.0 opam install core utop ocp-indent

6:03.75elapsed

So, 6 min to set up a dev environment with those tools. That installed 67 packages, so compilation took about 5 seconds/package on average. I'm not sure why it would build the same dep twice - I haven't seen it do that.

Yes, doing this would make it easier to use services like GitHub. For example, DataKit's GitHub bridge service uses webhooks to track changes in GitHub metadata (e.g. the status of branches, tags and PRs) and stores the state in another Git branch. The DataKitCI monitors this branch to know when to build things (and writes the status back there):

https://github.com/docker/datakit/tree/master/ci

Would be simpler if GitHub offered this data in Git format directly.

I use a Qubes Debian VM for dev work. Works fine, except that it runs gnome-keyring by default, which breaks ssh, and restores this config everytime I upgrade (but easy to fix when it happens).

Here's a common example. Some systems provide blocking operations: e.g. read_line waits for a line of text and then returns it as a string. Others use promises: read_line returns a promise of a string.

What if you want to write a library that works with either blocking calls or asynchronous promises?

The cohttp HTTP library is written that way. For example, the Transfer_io module[1] (supporting both chunking and non-chunking HTTP transfers) takes as an argument another module, IO[2], that provides a read_line function of type "ic -> string option t", where the type t is abstract and higher-kinded (and "ic" = input channel). You can instantiate the module with a blocking IO module (where a "string t" is just the same as a "string" and read_line blocks) or with a non-blocking one (where a "string t" is a promise for a "t" and read_line returns a promise).

[1] https://github.com/mirage/ocaml-cohttp/blob/master/lib/trans...

[2] https://github.com/mirage/ocaml-cohttp/blob/master/lib/s.mli

(also useful if your language has multiple competing promise libraries...)

I think this is what you're looking for:

https://github.com/docker/hyperkit

Source: https://blog.docker.com/2016/05/docker-unikernels-open-sourc...

Just to add some links for those (I assume these are the ones you mean):

Dumping the debug-level log ring-buffer on exception:

http://lists.xenproject.org/archives/html/mirageos-devel/201...

Detailed profiling (interactive JavaScript viewer for thread interactions):

https://mirage.io/wiki/profiling

It depends on which lightweight OS and which unikernel. But e.g. a stripped down Linux will still have a huge amount of C. If you're going to write your kernel in something safer, then you might as well make a unikernel, rather than creating a kernel/userspace split.

It's hard to see how you'd get a traditional OS stripped down anywhere close to e.g. the mirage-firewall unikernel (http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewal...)

Like most OCaml libraries, Irmin can be used in the browser by compiling your program with js_of_ocaml.

My `irmin-js` experiment provides a Javascript API to Irmin, so you can write your application in Javascript too, rather than OCaml. My JS isn't very good though; here's what my example code ended up looking like:

https://github.com/talex5/irmin-js/blob/master/examples/test...

Note: CueKeeper is a bit unusual here. Merges always succeed, but it adds a note to the item saying what it did to resolve the conflict.

e.g. if you rename an action "orig" to "a" and to "b" and then merge, you'll end up with an action called "a" with a note saying the change to "b" was discarded.

BTW: it's easier to generate merge conflicts using this web interface, which lets you run two instances in one window:

http://roscidus.com/blog/blog/2015/04/28/cuekeeper-gitting-t...

Mirage can be configured to write trace data to a ring buffer shared with another VM, where you can run a visualisation tool to explore it. For an example, see:

https://mirage.io/wiki/profiling

Of course, having such a small (and legacy-free) OS makes debugging vastly easier in the first place.

Are there any important changes to the system here, or is it just a new presentation?

After a few years using mGSD, I'm now using CueKeeper (http://roscidus.com/blog/blog/2015/04/28/cuekeeper-gitting-t...), which I wrote based on the first book. Any new ideas here I should incorporate?

I was thinking about smaller devices than the Pi. Security devices, monitors, IoT type stuff. Especially where you want to minimise the amount of C code running and know exactly what it's doing.

I think it's also fun from a hobbiest point of view to do useful things without a traditional OS (Xen still requires Linux or similar in Dom0), but instead using a safe high-level language for most of it.

This will be extremely useful for running on very small devices (not all ARM boards have the virtualisation extensions that are needed to run Mirage unikernels as Xen VMs).

But it also looks like lots of progress has been made with cross-compilation support in the OCaml compiler too. At the moment, I compile my Xen unikernels on my Cubieboard, but it would be quicker to cross-compile to ARM on my laptop and just push the final VM image.

I've been using fish for a few months now (switched from zsh). Really nice to have good defaults and minimal configuration, and the suggestions based on previously entered commands are great.

The main issues I have compared to zsh are:

- No bracketed paste mode. If I select multiple lines of text and middle-click-paste into zsh, it adds the whole lot as a multi-line input and lets me review the commands before pressing Return. fish just starts blindly executing them, which can cause accidents.

- Chaining commands is harder. In zsh `foo |& less` pipes stdout and stderr. Also, fish's `foo; and bar` is longer than `foo && bar`.

- Hashes in words start a comment in fish. e.g. `opam pin repo#branch` is treated as `opam pin repo`, pinning the wrong branch. zsh only treats # as a comment at the start of a new word.

- While completion history and `set -U` are really useful, it sometimes forgets everything and I have to start again, which is quite annoying.

On the whole, very happy with it though!

Note that the Lwt syntax extension (often used in Mirage) adds a try_lwt..finally construct. Doing anything at all with IO in Mirage will be asynchronous (there are no blocking operations anywhere).

However, people generally use various "with" functions to manage resources. That's better than try..finally because the allocation can't get separated from the try bit.

I never found Java checked exceptions much use for resource leaks, since unchecked exceptions are always still possible.

You're right. OCaml will throw an exception here. Many of the functions in the standard library throw exceptions when it would be better to return an option type (newer libraries are usually better).

The exception will be caught and logged in the "callback" function below. It then rethrows it, but the unikernel will continue serving requests.

Another option: if you're running Linux on the host, you can build everything there and then scp the unikernels into dom0 for deployment, which might be easier (that's what I do).

Not sure if you can build Xen unikernels natively under OS X though.

I sometimes hack on the C part (e.g. porting Mirage to ARM). The C code in Mirage is basically:

- malloc

- printk

- openlibm (standard maths functions)

- gmp (fast big ints for crypto stuff)

- garbage collector

- support code to make the above C code compile (assert, various standard header files, etc)

Once Rust has matured, and if reliable, well-optimised replacements for all these libraries become available, then it would be interesting to use it. But until then, there isn't much C code you can replace, because there is a minimal core you need to provide to compile C libraries like gmp, and we're pretty close to that already.

Rust's linear types / lifetime support is interesting, though. For example, when you pass a buffer to Xen for transmission on the network, it's important that you don't modify it again until it has been read (or you'll get a race). Rust's type system can express that statically, whereas OCaml's can't. On the other hand, OCaml has functors, is far more mature and has lots of libraries available.