Unix system programming in OCaml (2014)

https://ocaml.github.io/ocamlunix/index.html

by jxub • 8 years ago

269 48 8 years ago

ocaml.github.io

emmelaich• 8 years ago

Ocaml, in comparison to other functional statically typed languages has had many successful Unix applications.

For a great tour of going to Ocaml (from Python), see Thomas Leonard's blog.

A retrospective is here but read the whole lot.

http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret...

cbcoutinho• 8 years ago

And related hackernews discussion

https://news.ycombinator.com/item?id=7858276

a0• 8 years ago

This is such a great book. OCaml’s type system feels like a superpower specially in the context of Unix development which is traditionally done in C.

tntn• 8 years ago

Only somewhat related,but I've tried to learn ocaml before but have never been able to figure out when to put what kind of delimiters where. Can anyone recommend a resource that explains the usage of the double comma and the like?

laylomo2• 8 years ago

The double semicolon only matters in the REPL (aka the "toplevel").

Off the top of my head, I can think of at least two other situations where punctuation matters:

1) Whenever you have nested match cases:

    match left with
    | Some left ->
      match right with
      | Some right -> Some (left, right)
      | None -> None
    | None -> None

The compiler does not use indentation to figure out scoping, so the above gets treated as:

    match left with
    | Some left ->
      match right with
      | Some right -> Some (left, right)
      | None -> None
      | None -> None

So you need to surround the inner match with parens

    match left with
    | Some left ->
      (
        match right with
        | Some right -> Some (left, right)
        | None -> None
      )
    | None -> None

2) If you are doing imperative things inside an if statement:

Consider the following function:

    let cache_file file =
      if not (file_exists file) then
        printf "downloading file\n";
        download file
      else
        printf "file already exists\n"

The problem here is that OCaml allows one to create an if statement with no else clause. And since the compiler doesn't use indentation to figure out scoping, then the code is essentially treated like the following:

    let cache_file file =
      (if not (file_exists file) then printf "downloading file\n");
      download file;
      else (printf "file already exists\n");

Which is just totally wrong. So in order to do multiple imperative actions within an if block, you should use either parens, or begin/end delimiters (which are treated exactly the same as parens)

    let cache_file file =
      if not (file_exists file) then begin
        printf "downloading file\n";
        download file
      end else
        printf "file already exists\n"

dpwm5• 8 years ago

There's an interesting thing with the semicolon. I've heard people advocating thinking of the semicolon as a binary operator:

(;) : unit -> unit -> unit

Of course, the reality is more complicated than this, because it can be used at the end of an expression, where it behaves like a postfix operator of type 'a -> 'a. Let's take your example:

    match left with
    | Some left ->
      match right with
      | Some right -> Some (left, right)
      | None -> None
    | None -> None

you can actually do

    match left with
    | Some left ->
      match right with
      | Some right -> Some (left, right)
      | None -> None ; ; (* <-- two separate semicolons *)
    | None -> None

This doesn't just work for imperative stuff, like the operator theory of the semicolon, it will work with this example.

My understanding is that this is due to the fact that optional semicolons are allowed at the end of statements and close the current expression.

So we're closing the first expression (None) and then closing the second expression (the inner match). The double semicolon is something else altogether.

Though it is worth noting that this example is straying from idiomatic OCaml and we can avoid this altogether with better matching:

    match (left, right) with
    | (Some left, Some right) -> Some (left, right)
    | _ -> None

Both the native optimizing compiler and BuckleScript are able to do what you would expect with this: the matched tuple is optimized away.

Your second unfixed example actually causes a syntax error. There is an if expression in OCaml and an if statement. This is particularly unfortunate, because the let cache_file = function | file when file_exists file -> printf "downloading file\n"; download file | _ -> printf "file already exists\n" first semicolon within an if expression turns the whole thing into an if statement, and an if statement can have no else clause. The else clause is then without an if statement, because we've triggered the imperative if statement. This is a syntax error.

were we to define an operator

   let (>>>) a b = a; b

then we could replace that semicolon with an >>> and things behave as expected. But it's really that if can be both a statement and an expression. For this reason I prefer using matches in general.

    let cache_file = function
      | file when not (file_exists file) ->
        printf "downloading file\n";
        download file
      | _ -> printf "file already exists\n"

thedufer• 8 years ago

I can't think of any time a double-comma would be syntactically correct. Double-semicolons, which may be what you're referring to, are never required (although in some situations they will cause syntax errors to appear closer to the actual problem). They can be placed after all top-level value bindings.

Probably your best bet is to follow an intro series so you see some examples (https://dev.realworldocaml.org/ is the best at the moment, I believe), although if you're so inclined you could just read the BNF for the language (https://caml.inria.fr/pub/docs/manual-ocaml/language.html).

richeyryan• 8 years ago

Not a direct answer to your question but you could look at Reason. It is a C style syntax that layers on top of the OCaml semantics. Writing in Reason is writing OCaml. However, the new syntax makes it more approachable for people coming from C style languages and removes idiosyncrasies like the double semicolon.

Its still progressing and more focus has been given to getting it up and running in a compile to JS context but it should be equally as capable of systems work, especially as time goes on.

themckman• 8 years ago

For those interested, some of the underlying libraries that make up Docker for Mac (and, I think, Windows) are written in OCaml (or have components written in OCaml): VPNKit[0], DataKit[1] and HyperKit[2] (qcow2 support is implemented in OCaml).

0: https://github.com/moby/vpnkit

1: https://github.com/moby/datakit

2: https://github.com/moby/hyperkit

dfee• 8 years ago

To be fair, GitHub attributes 1.5% of hyperkit’s code to ocaml

djs55• 8 years ago

The 1.5% that's in the hyperkit repo is the shim to the Mirage block layer. The qcow2 implementation that's normally linked in is from https://github.com/mirage/ocaml-qcow .

pjmlp2• 8 years ago

They are taken from MirageOS.

https://blog.docker.com/2016/05/docker-unikernels-open-sourc...

rpcope1• 8 years ago

I think one major turn off, especially for systems programming, was the lack of multicore threading support (i.e. it had no way to get parallelism using threads). Does anyone know if this has changed?

atombender• 8 years ago

Posted a week ago: http://kcsrk.info/slides/mcocaml_gallium.pdf

Discussion: https://news.ycombinator.com/item?id=17416797

In short, expect it merged in mid-to-late 2019, maybe.

ZirconiumX• 8 years ago

While this is an issue, there are cooperative threading libraries (mainly Lwt and Async, though the community prefers Lwt) which can mitigate most of the reasons you'd need multithreading. Hell, because it's only single threaded, you can mostly ignore locking.

StreamBright• 8 years ago

Isn't this project aiming to solve that?

http://ocamllabs.io/doc/multicore.html

_0w8t• 8 years ago

One of the fastest web servers is nginx that uses child processes (typically one child per core) and asynchronous IO within the child without any threads. The same model with a parent process that monitor threadless child workers is often used in embedded system. One can and people do exactly the same in OCaml.

Surely it requires more code to setup things especially if the child processes need to communicate over shared memory to max the performance, but the big plus is that the resulting system is much more robust. In case of troubles one can just let a child die or even use kill -9 and start a new child. This is not possible with most threading implementations. For example, try to recover from out-of-memory in multi threaded C++/Java/Go etc. application. It is very hard. With threadless children it is almost trivial.

pjmlp2• 8 years ago

Plus we have learned the hard way that concurrency and processes are much saner from safety point of view than threads.

testaccount7• 8 years ago

IIRC in Coders at Work, Brendan Eich talks about hiring a programmer who wrote an OS in OCaml.

ofrzeta• 8 years ago

For some real world applications take a look at libguestfs http://libguestfs.org, a library that can inspect, mount, edit VM filesystem images. Also includes tools for p2v and v2v migration.

toolslive• 8 years ago

You really don't want to use the Unix module directly. If you're serious about system programming in OCaml, use Lwt or Async to allow for concurrency.

emersion4• 8 years ago

I'm contributing to a linker written in ~OCaml [1], and now I understand writing systems code in OCaml is really a bad idea. It makes things more complicated when you really don't want them to be. As James Mickens says [2]:

You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis.

[1] https://github.com/rems-project/linksem [2]: http://scholar.harvard.edu/files/mickens/files/thenightwatch...

phaer• 8 years ago

Can you provide an example of the things, OCaml makes more complicated and explain which languages you compare it to? I guess it's quite different in comparison to C, C++ or maybe Rust than to other garbage-collected 'system programming' languages like Go?

pasabagi3• 8 years ago

I think he's getting at the distinction between operational and denotational semantics. Presumably, operational semantics are a better fit for systems programming, because if you need to imagine what the computer is actually doing (Linus's comments about why he likes C comes to mind). Ocaml is more in a denotational direction than C - although to be honest, I don't ultimately understand the distinction, given that C relates to an abstract machine that is not all that similar to most machines today.

emersion4• 8 years ago

Here are a few examples:

- Inferred types: you're constantly wondering "what is the type of this variable?". Type errors occur far from where the real type error is. Annotating is complicated because you're often dealing with complicated types (with nested tuples for instance).

- Using higher-order functions: often you'll see a function with nested functions in it [1], or some functional list processing function like "unzip" being used [2]. This increases the code complexity a lot ("what does unzip do again?").

- Using monads: yet another distraction from what's really happening. It forces you to write nested functions and doesn't play well with conditions [3]. Also various people use monadic operators way weirder than >>= and all of this becomes really hard to read.

- Using functional-friendly data structures: the linker is using immutable lists of optional bytes to represent memory images, because this is idiomatic in OCaml. However this is slow and inefficient as hell. Really, I can't load a 60MiB binary because the linker runs out of memory and is killed by the kernel on machine with 6GiB of RAM. Also, just linking hello-world takes ages.

- Designing functional loops: when the loop you want isn't in your stdlib, you first need to create a recursive function, and then make sure all recursive calls are tail calls otherwise you blow up your stack. The result is usually not very fun to read.

tl;dr: the C language is small, the OCaml language is not, and when you're dealing with complicated stuff you really don't _need_ more complexity from the language itself. James Mickens explains this (among other things) in his essay.

[1] https://github.com/rems-project/linksem/blob/master/src/link... [2]: https://github.com/rems-project/linksem/blob/master/src/link... [3]: https://github.com/emersion/linksem/blob/54c3a8430e621198653...

(Note: set the syntax highlighting to "OCaml" when trying to read this source code)

EDIT: list formatting EDIT 2: grammar

rezeroed• 8 years ago

Aargh. Downvoting is so lazy. I'd be interested to hear both sides flesh this out. (Yes, I know the rule - don't complain about downvoting -- why do you think there has to be a rule about it‽)

dvfjsdhgfv• 8 years ago

The rule is ridiculous, but criticizing the rule is also forbidden.

Unix system programming in OCaml (2014)

Related Stories