The state of low-level system programming

http://erik-poupaert.blogspot.com/2012/12/the-state-of-low-level-system.html

by bluehex • 14 years ago

98 59 14 years ago

erik-poupaert.blogspot.com

bluehex3• 14 years ago

I really hope to see Go and Rust move in the direction of pluggable runtimes that can be embedded into a C host program. Currently, both runtimes require being seated as the main entry-point of the program to function. Both projects also have issues in their bug trackers to rectify this:

Rust issue: https://github.com/mozilla/rust/issues/3608

Go issue: http://code.google.com/p/go/issues/detail?id=2790

I wonder who will get there first.

PuerkitoBio• 14 years ago

Also, a way to compile without kernel (system) calls, so that it can be use for kernel development. It used to be possible with go, but no more with go1. Then these languages would truly be able to replace C.

rpearl• 14 years ago

C code can call into rust: http://static.rust-lang.org/doc/rust.html#extern-functions

bluehex3• 14 years ago

True, and so can C call back into go code with cgo. The problem is that it still requires that the main function be in a Rust/Go binary. A program written in C can't currently call into a library written in one of these two languages since the runtimes don't work unless they control the main entry point.

From the Rust doc linked: "The primary motivation of extern functions is to create callbacks for foreign functions that expect to receive function pointers."

jzwinck• 14 years ago

Then is it even possible to make a program that can use libraries written in Rust and Go and C++? If everyone wants to control main(), is there an escape hatch provided by any of these languages, like a C function that can be explicitly invoked near program start?

josephlord• 14 years ago

If you read what you linked to it says the main purpose is to provide C functions with callback arguments when called from Rust. It isn't clear from what you have written that this mechanism is available to generally access from C programs and scripting languages.

zem• 14 years ago

there's also ats [http://www.ats-lang.org/]. somewhere on my long-term todo list is to write a library of efficient data strucures in ats, to take advantage of some of its safety guarantees.

doublec2• 14 years ago

It's a pity ATS doesn't get more attention in the "Systems programming language" field. It can call into C, be called from C, provides a powerful type system, and can be used to write very low level code. I've written a few posts about how ATS can interact with C on my weblog here: http://www.bluishcoder.co.nz/tags/ats/

erichocean• 14 years ago

I like reading your blog posts on ATS, but I don't think I could use it effectively.

I've had much better luck with exhaustive testing with Klee and related libraries (e.g. for testing multi-threaded programs).

For whatever reason, they map better to how I think about problems, and I've found it easier to develop new code that is testable within those constraints.

zanny2• 14 years ago

Compatability with the C ABI I feel isn't a necesssity. On platforms built on C it is, but if you wanted to invent a new machine running a new system language with the ABI messes of C exonerated, you could. You would need to devise a translation layer in the OS executor to be able to interpret classic C ABI .so or .dll files for legacy applications and so you could compile C on these platforms, but that isn't a prohibitively hard challenge.

keypusher• 14 years ago

Doesn't almost all systems programming take place on platforms built on C? If creating a new machine and new language, convincing the existing systems programmers to move is going to take something very compelling.

davidw• 14 years ago

Historically, the "turtles all the way down" approach has not been very successful outside of niches. Lisp, Forth, Java have all been used for OS's that never seemed to go anywhere.

sparkie2• 14 years ago

The need for C isn't so much restricted by the operating system, but by the hardware manufacturers. The simple fact is that C is the defacto choice of language for hardware vendors to ship a compiler for, and that doesn't look to change any time. It's a standard that can't be avoided, unless we can work cooperatively with enough major vendors to change this.

TheCondor• 14 years ago

There is C specific hardware?!?

This is a myth and part of the vicious circle here. I'm not a C hater by any means but the reason hardware vendors have historically included it is because there are free compilers that are nearly C enough that they can include and they have to include something. C is just a portable assembly, you can almost write a simplistic compiler in pure yacc and more than a couple of these C compilers hardware vendors produced weren't much more than that. Software guys then wrote in C and the circle continues, intel has a great C compiler now and in fact all the best optimizers are from hardware vendors but that wasn't always the case. There was also a time (quite possibly it's still that time) when you'd buy some customed up MIPS or PowerPC chip and get an ancient version of gcc that they'd hacked to support their shit and forward porting to a modern GCC wasn't always a straight forward thing to do. They have to provide something and tools usually isn't their business.

Now so much is built on the C non-runtime that Go, Ada, Rust, anything with any runtime looks completely backwards to a lot of system folk. What's funny, you look at Linux and there is a fairly small hardware dependent directory of code where the register poking happens, it's really not that much code but its called from all the drivers and such, and that's the holdup. It's not like that stuff makes up 50% of the code or anything.

gmt20272• 14 years ago

Vala does not seem to get much attention. It borrows C# syntax (and maybe Java), compiles down to C and offers nearly symmetrical interoperability as a result. I believe Ubuntu's Unity is written in Vala. Knowing Java, it was easy to acquire the syntax from the tutorial [https://live.gnome.org/Vala/Tutorial] in very little time.

For transparent access to compiled C libraries, external C functions need to be declared in a 'vapi' file and although these were already provided for a large part of the GNOME ecosystem, documentation on vapi syntax was nearly non-existent. Object-orientation and standard libraries are integrated with Gnome's GObject system and Glib libraries which I found unfamiliar and usually better documented in C at the time.

qxcv• 14 years ago

If it has GObject as a dependency, then it's not designed for "low-level systems programming". Hell, just having a GC more-or-less discounts it due to the associated problems with runtime size and performance[0]. As a general rule of thumb: if you can't write a kernel in it, it's not a systems language.

[0] It is possible to write a systems language with these features, but low-level systems stuff is not what Vala was designed for.

gmt20272• 14 years ago

GObject itself is not a dependency, Inheriting from GLib.GObject provides support for features like introspection and better integration with the collections framework and GLib libraries. However, lightweight 'non-object classes' are supported but treated differently by the compiler. There is a slight variation in constructor syntax between plain objects and GObjects.

As far as I know there is no Vala runtime or garbage collector. Memory management is by automatic reference counting but there is support for manual C++-style pointer syntax with new and delete operators.

I agree that Vala was not specifically designed for 'low-level systems stuff' but I see no reason why this is not technically possible. The Vala compiler is a source-to-source translator to C and in my previous experience worked fairly well interfacing with fairly low-level C libraries like MPI and OpenCL. I assume that a kernel writer does not require much more than a limited subset of the language that supports C pointers, structs and arrays.

justincormack3• 14 years ago

The C ABI is not great for interacting with scripting languages though, even if it is the best we have right now. For a start there is no type information obtained with dlsym and you basically have to parse headers the interpretation of which is not clearly defined as the C language really defines an API and the ABI is defined by the compiler. Many libraries for this reason do not define stable ABIs and are a pain to use from a scripting language. Hence Python has two ffi modes an ABI interface and a compiler mode.

Now if someone produced a better solution to that it would be useful...

qznc2• 14 years ago

In practice it works quite well, if you restrict the ABI to function calls and word-sized values (long and void). It especially means not to use structs. Instead you need void and getter/setter functions.

justincormack3• 14 years ago

Thats very limiting. A lot of applications need arrays (and maybe complex numbers) eg for numerical applications. And it is hard to manage without some sort of int64 type nowadays.

And also say you want performance, then using getters and setters sucks compared to direct struct access, as every access needs a non inlineable function call.

angersock• 14 years ago

You know, it's strange, because the Lua folks seem to be getting along just fine.

justincormack3• 14 years ago

Yeah but us Lua folks are picky about the sort of C we like to interface to.

tehwalrus• 14 years ago

I asked a very relevant question on stackoverflow about this recently: "Writing a Python extension in Go" http://stackoverflow.com/questions/12443203/writing-a-python...

Now technically you can compile a special python which can handle Go extensions, which is the wrapper mentioned in the second answer, but you can't distribute python modules that ask people to install "this special python" to run (which will reliably work with all their other installed code..)

I would like to see a "go init" function in a C API, that would let you interop the two whichever had the `main()`, since that would let you link with all the other c-implemented scripting languages (python, ruby, etc) which would mean that Go extension support could be added to the main cpython trunk as a compile-time option.

damian2000• 14 years ago

If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already.

I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++.

pretoriusB4• 14 years ago

>I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++.

Google at large barely uses the language. All the examples they've given are of peripheral stuff, like a load balancer for MySQl for YouTube and an even simpler replacement for an aged and cranky C++ system for Google Downloads. Nothing to write home about, and no replacing C++/Java for core systems at all.

slurry• 14 years ago

It seems to me that they are, in fact, using "statically-typed, compiled system language[s] for their scripting efforts" - that is in fact what people are doing with server-side Java, Scala and C#, not to mention Java/Dalvik and Objective-C on mobile platforms.

Perhaps this article misses the point a bit. For several decades we have been using scripting languages to glue together sharp pointy C[++] tools (sh/Perl/Python/Ruby approach) or to puppetize monolithic C[++] hulks from within (VB/elisp approach).

More recent practice, on both mobile and server platforms, has been to abstract away from C[++] with a middle layer of memory-managed, compiled C-ish goop - JVM, .Net, Objective-C, whatever. This is the level where people are doing everything from amazing high-availability big data things to amazing little bleep-bloop handheld things. I think the answer to "what's the state of low-level systems programming?" is "not much is going on, but damn, there's a lot of fresh stuff happening in mid-level systems programming."

Go and Rust are late entrants to this party, and probably exist more as a hedge against Oracle litigating Java/JVM into the ground than anything else.

cturner• 14 years ago

I don't think this stands up: "It must be possible for such contender to be called from C code, because that is what it takes to be called from a modern scripting engine too."

Later the author says, "If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already."

I think the reason that scripting languages exist is because of limitations in the established languages. What we think of as "a scripting language" is defined by the gap. If someone else could be a system programming language without the gap - maybe it doesn't need to be accessed.

The traditional unix platform is (1) C and a compiler for it, with access to the unix standard library and (2) some sh or csh derivative, and access to the likes of sed and awk.

Shell gave people a glimpse of what was possible. But shell has poor access to the unix API, and hits the wall quickly once you do anything other than very basic tool development.

So these things called scripting languages started to pop up. People wanted easy access to the unix API like C. They want easy string manipulation like shell. And they want access to functions, maps and lists without having to think about memory allocation.

If you start from the shell and awk and work into that gap you get perl - the canonical scripting languge.

TCL inhabits a similar space to perl. It's also "a scripting language". But it started at the opposite end. TCL starts as an extension language for C.

So I can understand the author's emphsais on close interaction with C. But I don't think they're important if you can take away the shortfalls they're compensating for.

I hope Go will mean I don't have to think about these things. That I'll be able to get a standard build and a go compiler/runtime and have immediate access to lots of power. No more headaches from C build tools or reaching around in the murky water underneath my scripting language.

jedbrown• 14 years ago

You are leaving out the importance of libraries and coupling of systems that were initially written independently. This is the same mistake that many Lisps have made, assuming that just because the technology is "superior", people will flock to it and eagerly rewrite all their code (starting with their dependencies).

pretoriusB4• 14 years ago

>The Go language and the Rust language. What they have in common, though, is that both Go and Rust are entirely incapable of walking like a duck. Since you cannot call Rust -or Go code from a scripting engine, they are inadvertently positioning themselves as alternatives to the scripting engines themselves.

Before writing all these BS, maybe he should do a little research first?

Here's Rust (which is not even released yet) being capable of being called from C. It's a feature they are actively work on:

https://github.com/mozilla/rust/issues/1732

bluehex3• 14 years ago

> Before writing all these BS, maybe he should do a little research first?

There's a subtle difference between being able to call back into Go/Rust code from C (where the main function is in Go/Rust) which both languages allow now, and allowing a call to originate from a C program, which neither currently do. The author is talking about the later.

I would agree the article could go into more detail about what's currently possible, and why it's not enough. But calling BS, I think, is going a bit too far.

pretoriusB4• 14 years ago

>and allowing a call to originate from a C program, which neither currently do.

That one is also under consideration by the Rust developers.

groovy2shoes• 14 years ago

Clay[1] doesn't get as much attention as it deserves. The authors seem well-versed in programming language theory, but they also understand the practical needs of a systems language. What you end up with is a language that's sufficiently low-level to be a C replacement, but it offers a plethora of mechanisms for abstraction.

Of all the new languages billed as "systems languages", Clay is the only one that excites me.

It can also "quack like a duck" (to put it in TFA terms).

[1] http://claylabs.com/clay/

MatthewPhillips2• 14 years ago

I can't find any kind of getting started guide, and most of the libs have no documentation at all.

qznc2• 14 years ago

I found a "Clay for C++ programmers" tutorial

https://github.com/jckarter/clay/wiki/Clay-for-C---programme...

ponce• 14 years ago

May I mention Nimrod [1]? It has a lot going for it when it comes to meta-programming.

[1] http://nimrod-code.org/

MatthewPhillips2• 14 years ago

So what's the catch when it comes to D? If I can use all C libs easily, and get some more palpable syntax, what am I missing exactly and why haven't people transitioned to it?

pretoriusB4• 14 years ago

The article is not about using C libs, is about C programs using you (your D code, that is).

That said, the problem with D: not enough marketing plus the two compilers / 2 libraries issue.

And it's close to C++ complexity enough to not let the scripting guys feel welcome (like they do in Go).

frujka• 14 years ago

it's probably an interesting read except that being hosted on google's blogspot means it only display blank pages due to content being embedded in some javascript shenanigans.

Even a screenshot would be better than this.

Please move on to some actual html website that just works or rehost the content elsewhere before posting it here. Thanks.

eropple• 14 years ago

I used to be one of These Guys who never used JavaScript, too, but honestly? It's 2012 and JavaScript is a thing pretty much everywhere worth targeting. Demanding that somebody cater to your microsegment is a little much.

vq• 14 years ago

Although I'm the kind of guy who usually have JS disabled I could be made to see your point. But this page is actually a good example of why I run with JS disabled. It magically positioned links and boxes over the scrolling bar and caused my initial attempt at scrolling down to navigate away from the page.

Disabling JS usually have the effect of doing away with such nonsense but here that is no option.

This time it was worth the hassle since it was an interesting article, but I definitely prefer it when JS is optional.

nnq• 14 years ago

> rehost the content elsewhere before posting it here

Google's blogspot allows you to do normal html blogs (maybe now they've made this their "default", but still, it's OP's choice to have it this way)

sublimit• 14 years ago

My attempts to read this on my phone were futile. Every time I tried to scroll, it just jumped back to the top immediately.

dschiptsov3• 14 years ago

This is called FFI.)

And yes, a language with which you cannot dload and call any shared library you wish is, how to say, handicapped language.

NodeJS, Java to name a few. Look what they do due to inability to call, say, libpq.so.)

But of course, re-monkey-patching everything from scratch by amateurs coders is much better idea. And, you know, you could create an entire "standard" with hundreds of pages of specification and then push your bloatware - JDBC.

pjscott• 14 years ago

I believe his thesis was actually that most languages are handicapped because they can not easily be called via the usual "load a .so file and call a function" ABI. Java, for instance, would be a more useful language to me if I could easily create Python bindings for useful Java libraries like Lucene, without needing to use something like Jython that runs on the JVM.

amirouche• 14 years ago

tl;dr: use JNI through pyjnius of jpype.

You can run Python 2.5 on lastest Jython, then you have access to JVM libs and the JVM libs to Python libs 2.5 compatible.

Calling JVM from CPython is actually possible: - with pyjnius from kivy fame built with cython it is still in development, AFAIK it is missing the possibility to call Python code from Java code aka. submitting a Python "callback" to a Java method, I think it is called IoC pattern and Hollywood principle «Don't call me, I call you». jnius is used in Kivy to make it possible to call Android libs. As an example, I created Python bindings of Blueprints with it. Blueprints is an abstraction library for graphdb like Neo4j, OrientDB, Titan, available here https://github.com/Printemps/python-blueprints/. Tutorial: «JavaClass = autoclass('path.to.JavaClass'); JavaClass.JavaMethod(python_object)». Those bindings are merely Pythonification but they are gotchas in types conversions.

- There is also jpype which looks like an established solution used by Neo4J fame to make it possible to embed Neo4J embedded into CPython.

I choosed jnius because it's faster; Those are funny benchmarks in log scale: https://docs.google.com/spreadsheet/ccc?key=0Ajc7MnpSjCokdGt...

dschiptsov3• 14 years ago

The confusion arises when some language developers decide to ignore or break existing ABI for a particular platform.

There are general ways to call procedures - there machine instructions for this. ABI defines how exactly parameters and return values should be passed - using, registers or stack or both.

As long as you follow the rules there is no difficulty in calling everything you wish.

The whole idea of JVM as something disconnected from reality^W OS and hardware is simply wrong, and all the confusion is the consequence of that premature decision.

Of course, slogan "you don't have to know" always wins.

guard-of-terra• 14 years ago

I'm so happy Java can't load libpq you won't believe. Because it means that Postgresql API gets reimplemented as a crispy JVM code that never crashes my JVM. If we would live in the world where a Java program would use libpq and libxml and libldap and whatever, it would be so much less stable, introspective and consistent that it won't be Java anymore. It would be a particularly ugly kind of Python.

And yes, it would crash when you use threads. Since all those C libraries pretend that threads don't matter. Guess what, our Java program has 500 threads and feels fine.

dschiptsov3• 14 years ago

Technically, an instance of JVM is mere a user process, which, by definition, must obey "laws" for user processes. This could be end of story - the idea that something "reliable" could be implemented as a single user level process is, well, naive.

btw, you still must call libc.so and libpthread.so

As for pthreads, guess what, they are also bad decision, as long as they are not isolated as processes supposed to be, and crash of one thread messes up rest 499.

People who did some research, like developers of Erlang, found that neither one-process model or threaded-threaded process model is even applicable when you need a reliable system.

So, again, jvm.exe is mere yet another .exe.)

Yttrill2• 14 years ago

Au contrare it is C which is handicapped.

How handicapped is C++ considering it subsumes C, can use C right at the source level, has an inline FFI, and yet C++ libraries can't be called by C because C++ provides type-safe linkage.

Advanced languages are advanced because they improve on C and this inevitably means they must go beyond C. This is why such languages require setting up an environment, usually by providing main().

Even if you could set up an environment with a C function call, which is trivially achieved by replacing main with mymain(), you would simply be starting a foreign environment, not providing a library of functions which could be individually called by C.

My system Felix (http://felix-lang.org) is specifically designed to live with the compromises of C/C++ compatibility, but you still cannot put arbitrary Felix code in C callable functions because Felix provides an advanced environment supporting, for example, fibres with channels. Fibres are scheduled by returning control to a scheduler so they cannot be nested inside a function which puts its return address on the machine stack.

If you Felix write code without using features requiring advanced support that code can be made into a C callable library (both static and dynamic linking is supported).

Felix is probably better at C/C++ integration than any other language except C++.

pcwalton• 14 years ago

Freestanding, ABI-level interoperability with C is planned for Rust, but is not yet implemented. There are two main reasons why being able to be called with dlsym() was less important for us than the current setup:

1. M:N scheduling. We wanted high scalability with hundreds of thousands or millions of tasks, which means we need the capability to have our own scheduler. OS threads aren't lightweight enough for this, especially on Windows, where over 80% of our Firefox users are. Having our own scheduler means we need our own ABI. We would like to make the scheduler optional, but getting it working was a higher priority than making it optional.

2. Safety. On many platforms (basically, anything but MSVC on Windows), the C stack is not bounds-checked; either there is no guard page or the guard page is unreliable because you can overshoot it. This is not acceptable for our Rust code, since stack overflows are a frequent source of exploitable security vulnerabilities for us. So we do our own stack management with a custom ABI that performs stack checks at every function entry, freeing us from this source of pain on all operating systems. Again, this is planned to be optional, but it was more important at this early stage for us to demonstrate advantages over C than for us to duplicate its ABI exactly.

Many libraries, however, would prefer direct interoperability with C to M:N scheduling or stack overflow safety. So we plan to provide a mechanism for Rust code to be called from C without having to set up a scheduler or perform stack checks. It's not implemented yet, but it's very much part of the current design, and I would be thrilled to see it happen.

lispertoascheme• 14 years ago

The state of programmers trying to avoid coding in the language their system is coded in. C.

martincmartin• 14 years ago

For example, Java and C# cannot walk like a duck, not even if that is what it would take to save themselves from drowning. Therefore, they are absolutely unsuitable as system languages.

Hadoop held the world record for sorting a Terrabyte for a while. If sorting large amounts of data as fast as possible isn't a system's programming task, then I'm a three-headed monkey.

The state of low-level system programming

Related Stories