The Unix C library API can only be reliably used from C

https://utcc.utoronto.ca/~cks/space/blog/unix/CLibraryAPIRequiresC

by jandeboevrie • 7 years ago

91 70 7 years ago

utcc.utoronto.ca

Eikon• 7 years ago

There is no useful errno variable to load in your own language's runtime after a call to, say, the open() function

FWIW, in rust there is std::io::last_os_error().

libc::perror works too.

x3ro• 7 years ago

The point of the article is that there's glue code needed for this, I think.

It seems like Rust also needs this, either through linker magic or extern C:

https://github.com/rust-lang/rust/search?utf8=%E2%9C%93&q=er...

miohtama• 7 years ago

The whole C interface is clunky because C does not support multiple return values. More advanced languages should not be forced to deal with it.

smitty1e2• 7 years ago

"Should" is as wobbly as errno, no?

masklinn3• 7 years ago

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

guerrilla• 7 years ago

Are you sure its not the other way around? Might C not have return values because of the systems it ran on? How would that work today in Linux, for example? Map a page to have some memory shared between the kernel and userspace?

masklinn3• 7 years ago

The whole C interface is clunky because C does not support multiple return values.

C has pointers which can easily enough be used as "out parameteters". In fact the C standard library regularly makes use of these. It's not amazing, but it works fine, and it's usually fairly easy to go back and forth e.g. expressing a stat(2) call is reasonably straightforward.

jononor• 7 years ago

However there is no way to know that programatically, and no standard way to annotate it either. So in FFI / language bindings context this is really tedious to work with.

merb• 7 years ago

well most of the time the return value is just an error integer, while the real return value is a pointer parameter.

moomin• 7 years ago

My hot take here is that multiple return values are a conceptually complex way of handling tuple return values.

The C API could do this, it just doesn’t.

rwmj2• 7 years ago

You can return structs from C functions (note, I don't mean pointers, I mean whole structs). You can pass in structs too. It's not commonly done and is a bit clunky compared to returning a tuple, but it is possible.

chewxy• 7 years ago

What's the difference? My understanding is as follows: A struct is a tuple. It's just some structured memory. Where a tuple would be indexed numerically, a struct is indexed with names of the fields (equiv in Python to NamedTuple)

masklinn3• 7 years ago

It's not commonly done

It’s really rather common eg several time or socket functions take a pointer as « output parameter », fill it and return an error code.

Not the best but not the worst either.

justincormack• 7 years ago

The Unix API even returned a pair of file descriptors for the pipe syscall. Most ISAs have a way of returning multiple values eg for structs by value. However they fell out of fashion, I think only NetBSD still has a multiple return pipe on some architectures.

mannykannot• 7 years ago

The issue here is the interface to the operating system, which should be as universal as is feasible. Having multiple return values would work against that, just as using macros does.

zozbot234• 7 years ago

Back in the day, other languages had their own bindings to the POSIX/UNIX API's. There are standardized bindings for FORTRAN and Ada; not sure if others exist.

Too• 7 years ago

This is a fairly common problem when trying to interface other languages with any C API, not just errno. The FFI-library of your language can usually call straight into C-functions but then it turns out half of the C API is implemented in macros which can't be called nor converted automatically, so you have to be re-implement them all in the new language.

Funny because usually the reason for having a C API is to provide universal compatibility to be called from any language. If you are doing this, please avoid macros.

arnsholt2• 7 years ago

I ran into that at work a few years back. I was tasked with integrating OpenSSL into the commercial Smalltalk used for the project (last updated in 1999, for more or less good reasons). A priori, not an unreasonable task as the Smalltalk had a pretty good FFI mechanism. Except of course the neverending cavalcade of functions documented in the OpenSSL docs that were actually preprocessor macros. In the end I wrote a Python script that parsed the C headers and generated FFI entry points for normal functions and Smalltalk code for preprocessor macros that weren't simple literals. The result was a small mountain of code, but it made all of OpenSSL available to our application, which was great.

wruza2• 7 years ago

You’re still lucky it wasn’t like DirectDraw or something WinAPI in general. I had to reinvent entire preprocessors and investigate implicit compiler defaults to be able to wrap that, and still there was LOWORD, MAKE_XYZ, etc, which were official arbitrary code-in-a-header interfaces. Preprocessor is a scourge of an API.

jstimpfle11• 7 years ago

When I worked on my own language that links to libc, it turned out that MSVC has the standard streams (stdin etc) defined as macros. That's really annoying since that means one has to write a C wrapper function to expose these streams. It might have been possible to avoid these macros, but that probably puts some constraints on the allocation done inside libc.

On the other hand, there are some things you simply cannot do without macros. I usually use function-like macros to forward to other functions with additional arguments (like __FILE__, or #arg, sizeof(arg)) which is extremely convenient and allows to cut out a lot of error-prone boilerplate.

From a different language, you can still call the function that is being forwarded to. Now try doing that with C++ templates or whatever...

jmgao2• 7 years ago

Hilariously, the C standard requires this.

C11 7.21.1.3: "The macros are [...] stderr, stdin, stdout which are expressions of type ‘‘pointer to FILE’’ that point to the FILE objects associated, respectively, with the standard error, input, and output streams."

varjag• 7 years ago

Looking at you, PJSIP!

pjmlp8• 7 years ago

Which is also kind of ironic, because C as universal compatibility only works on OSes implemented in C, as it is the OS ABI, which granted is the case being discussed here.

Someone2• 7 years ago

"as it is the OS ABI"

The C ABI almost never is the OS ABI. On systems with separate address spaces, the OS ABI needs to cross a kernel boundary. In theory, you could use the same ABI for regular function calls, but the overhead wouldn’t make your compiler popular.

coldtea• 7 years ago

The FFI-library of your language can usually call straight into C-functions but then it turns out half of the C API is implemented in macros which can't be called nor converted automatically

Why is that? Aren't the macros expanded by the preprocessor, and end up as a specific piece of code under a function name in a library?

Or does this mean there's no e.g. X function that uses one or more preprocessor macros inside it, but the very X function is but a macro itself (so it's only "callable" by the public API function name, from C code before expansion)?

buckminster• 7 years ago

Aren't the macros expanded by the preprocessor, and end up as a specific piece of code under a function name in a library?

Sure, but you don't know the name of that function so you can't write the FFI declaration for it. You could reverse engineer it, but this would be platform and compiler specific, so non-portable and unstable.

yxhuvud• 7 years ago

Another thing to avoid is functions defined in the headers as `static inline`.

GlitchMr• 7 years ago

A function internally called by `errno` macro is part of an ABI, and changing its name will break all already compiled applications for no real reason. Most operating systems provide stable libc ABI (exception: OpenBSD). So it's a rather safe thing to depend on.

ainar-g2• 7 years ago

(exceptions: FreeBSD, OpenBSD)

These are some pretty significant exceptions though, especially in the networking-related fields. I don't want the programmes on my router to suddenly not boot after a system upgrade. Especially when I don't have the source code or if the compilers haven't caught up yet.

rkangel• 7 years ago

The issue here is that you shouldn't need to interface with C, unless you need to talk to some C Code. You absolutely shouldn't have to interface with C to talk to the kernel.

I come at this from a Linux perspective, but to me the user/kernel interface (sys calls etc.) should be the well defined abstraction boundary. The C library is a useful api making that easy to use, but to dictate that people can only make sys calls through C is silly and ends up with these sort of problems.

amluto2• 7 years ago

There is quite a bit of variation here. Linux and, AIUI, quite a few microkernels, have a defined ABI at the user/kernel boundary. BIOS, a bunch of hypervisors, and some RISC-V M-mode interfaces are similar, albeit with “user” and “kernel” replaced with other execution modes.

OpenBSD and, very notably, Windows instead have a defined library interface. Windows has somewhat stable “system calls”, but only through NTDLL.dll. The real stable interface is kernel32, etc.

There are arguments in favor of both approaches. One significant, if rarely appreciated benefit of the user/kernel approach is that it enables user programs that don’t use the C ABI at all. Go on Linux is like this, and Go and the vDSO have interoperability issues because the vDSO is a bit FFI-ish in Go, and Go’s FFI is not very fast.

the_why_of_y• 7 years ago

A stable syscall ABI is a Linux peculiarity and was never a part of the UNIX tradition.

Here's a talk where Bryan Cantrill describes the tightly coupled Solaris kernel/libc combination as a "welded unit", and mocks the Linux "libc ecosystem" and the complexity of ENOSYS handling in glibc.

https://www.youtube.com/watch?v=TrfD3pC0VSs#t=29m10

wsxcde• 7 years ago

There's no problem with directly calling sycalls from OpenBSD from golang or C or whatever else. The issue here is that OpenBSD is adding a new security feature where code addresses that call syscalls must be whitelisted at link time. golang could just create their own syscall interface and linker scripts appropriately whitelisting these locations and this problem wouldn't exist.

matheusmoreira• 7 years ago

This happens due to POSIX. Nothing stops people from using direct system calls but they would rather use a C library for portability reasons.

I'd even say the Linux system call interface is better than the POSIX one. Well documented, no errno. I wrote a liblinux in C and Rust and it's been useful for accessing newer system calls with no libc stubs. The more subtle problem: standard C libraries are incompatible with direct system calls. They maintain a lot of global state which will be invalidated if the program calls the kernel directly. For example, clone() cannot be used with glibc.

The ideal solution is to start from scratch and build a new user space without using C at all.

xvilka• 7 years ago

I think it is time to create the new POSIX standard, with better portability and safety/security. Decades of experience can help to design the better API.

ailideex• 7 years ago

New POSIX standards are released quite frequently and participation is quite open, so ... have at it :)

pjmlp8• 7 years ago

It will never happen, because POSIX is literally the UNIX part of C libraries that did not land as part of the ISO C standard library.

touisteur• 7 years ago

One of the worst offenders is statvfs with its use of __USE_FILE_OFFSET64...

skissane• 7 years ago

The complaint is that the underlying function which returns the thread-local errno pointer isn’t a public API, isn’t standardised, and its name varies from OS to OS (__errno or __error or __errno_location)

This specific case would be resolved if some standard (ISO C, POSIX, etc) defined a public API to do it, e.g. errno_r

(Of course, that doesn’t fix any other cases where some public functions are really macros to private functions, but this appears to be the most significant such case.)

The Unix C library API can only be reliably used from C

Related Stories