Making Win32 APIs More Accessible to More Languages

https://blogs.windows.com/windowsdeveloper/2021/01/21/making-win32-apis-more-accessible-to-more-languages/

by ingve • 6 years ago

159 49 6 years ago

blogs.windows.com

mappu• 5 years ago

Golang is also considering consuming this new metadata: https://github.com/golang/go/issues/43838

sudeepj• 5 years ago

But golang's CGO does not link with msvc compiler right?

Are they planning to change this?

ChrisSD4• 5 years ago

You don't need the msvc compiler to make use of the metadata. You just need something that can read the winmd format.

wruza• 5 years ago

A good thing that is late for 25 years. Still better than never, WinAPIs never were an easiest thing to wrap.

emddudley• 5 years ago

Rust for Windows uses this. https://news.ycombinator.com/item?id=25862291

userbinator• 5 years ago

There was some discussion on this earlier today: https://news.ycombinator.com/item?id=25885196

jiggawatts3• 5 years ago

One issue I've had with any kind of auto-generated bindings based on metadata is that you get low-level types such as "Win32 BOOL" and "Win32 GUID" polluting the output. Mainstream languages already either have native bool and guid types, or have defacto standard libraries.

E.g.: Rust has the UUID crate: https://crates.io/crates/uuid

Similarly, Java has had a UUID type since v1.5: https://docs.oracle.com/javase/8/docs/api/java/util/UUID.htm...

It's not just concrete types, but core language concepts often "don't translate". E.g.: Win32 is essentially a C-like set of bindings. In many languages such as C++, D, Zig, etc... this is not an issue. However the type system of C is much less strict that Rust, so the metadata has no concept of ownership and lifetimes. This means that if you want to consume Win32 APIs from Rust, you're still forced to wrap everything manually at least once to escape the "unsafe" sandpit.

A thought I had is that C is used for API definitions because "it is lowest common denominator". But that's the wrong thing to do, the metadata doesn't have to "sink to the lowest level"! An ideal metadata definition would be something akin to Rust's type system, or even stronger than that! "Degrading" a binding from a strong type system to a weak one is trivial and can be performed completely automatically. The converse is difficult and error-prone even for humans, and impossible for code generators. You can't synthesise this missing data!

Even the read-only guarantee is too weak. E.g.:

    // Rust could treat 'buf' as read-only borrow.
    void SendData( [readonly] void* buf );

    // But here the borrow outlives the function call lifetime!
    void SendDataAsync( [readonly]void* buf );

The two methods have identical metadata signatures, but the information required by a Rust programmer or a Rust code generator is missing...

There are many such abstract API concepts missing from C, such as thread safety, process integrity safety, non-returning calls, blocking vs async, etc...

brundolf• 5 years ago

One issue I've had with any kind of auto-generated bindings based on metadata is that you get low-level types such as "Win32 BOOL" and "Win32 GUID" polluting the output. Mainstream languages already either have native bool and guid types, or have defacto standard libraries.

This shouldn't actually be a huge problem in Rust, because of the traits system. It should be possible to implement all the expected behaviors (Into/From, Add/Sub/Mul/Div, Ord/Eq, etc) for these special types and if library code is properly generic, it shouldn't care whether or not you're using the native type. If it does require a specific concrete type, From/Into should still make things reasonably ergonomic.

Re: safety, the typical approach has just been to keep one library that's a direct mapping to the C API, which is publicly unsafe, and then create a safe library abstracted on top of that which codifies the documented behaviors. That layer relies on documentation, and on humans to follow it, but then so does all C code ever written :)

jiggawatts3• 5 years ago

In my experience with both classic Win32 programming in the 1990s in C++ and then later in C# and now in Rust is that it is obscenely verbose. It takes a solid page of code to call a single method, and most of that is just noise.

A typical method call is:

    Allocate some struct.
    Zero it out.
    Set a "length" field to the length of the struct.
    Populate the "parameter" fields.
    Call the API with the struct and 0 for the return buffer size.
    You get back an expected error code for "buffer too small".
    You allocate the required space in your buffer.
    Call the method again. The amount of data changed! Loop and try again.[1]
    You get your data! Now to check the error codes.
    Look up error code using a Win32 method.
    Free the buffer.

If on top of that I'd have to call "into" and "from" for every field and every parameter at every step... I think I might just snap and go become a bricklayer or something.

PS: Win32 strings and Rust strings aren't compatible. I've already noticed that the current version of the Microsoft code-generator seems to use an awful lot of [u16] arrays, which are not the system OsString struct.

For comparison, the C# "String" type isn't directly compatible with Win32 strings either, but the FFI engine handles this conversion back-and-forth for you.

Similarly, few (any?) languages have good built-in support for the various kinds of "struct-followed-by-dynamic-data" typically returned by system APIs. Notably, C does not have any way to encode this in the type system, so API metadata tends to just leave this out or describe it in the documentation. If the metadata was extented to cover this kind of thing, then the generators could do all sorts of clever things with this scenario...

[1] A lot of people don't do this, they assume that the second call will always succeed. Programming in this style leads to rare random errors throughout the entire application. A loop is required for correctness for most (but not all) Win32 calls that return dynamic data...

ChrisSD4• 5 years ago

A Rust `bool` is not a WinAPI `BOOL`, for one thing they have different sizes. A wrapper function could translate between the two... but even that's not universally possible. Depending on the API, `BOOL` can sometimes have three or more values (e.g. -1, 0 or 1). So whatever happens you need a safe wrapper around the function call that understands what `BOOL` means in this specific context.

Incidentally, there is no reason why the metadata can't be improved with more information that can be used to create safer wrappers. The winmd format isn't restricted to only that which can be represented in C's type system.

DaiPlusPlus3• 5 years ago

winmd is limited to the .NET type system - an improvement over C, but still insufficiently expressive enough imo.

I suppose you could hack it by adding new Attributes...

hctaw• 5 years ago

There has to be a bottom to the stack. What I mean is, at some point types become bytes and instructions. Not every program or interaction is capable of being expressed at every levels of that stack, the invariants must be broken in code and maintained explicitly by programmers on either side of APIs. That simple fact is why FFI boundaries are inherently unsafe.

This doesn't have to do with C, it has to do with the nature of systems programming - not everything can be baked into the one type system to rule them all that is checked by all programs in all languages at compile time. Metadata is insufficient, you need runtime type checking, which is not free.

And to be honest I don't think this is an attractive thing to strive for. OS APIs are not that hard to program against (well, maybe Windows APIs are). Differences between OS's and versions are, however. We don't really need safer systems interfaces, what we need are standard ones that actually work to solve hard systems problems like asynchronous i/o, real time scheduling on non-realtime kernels, and arbitrary control flow. These are very unsafe things that do not have good solutions in the OS APIs, adding strict types where they are not needed creates little value for the systems programmers that want to make cool new systems.

withinboredom• 5 years ago

So much this. People (and languages) seem to forget that an HTTP request is a string of bytes (another “bottom of the stack”). A JSON object isn’t a type, it’s a string of bytes.

flohofwoe2• 5 years ago

I'm tackling this issue from two sides:

(1) Change the C-API to make it more "binding-generator-friendly", for instance by adding a range/slice-struct to th C-API which bundles a pointer and associated size, or specially named typedefs that only exist to give the binding generator hints for special case handling. By adding a C-type which represents a range/slice, it's easier to connect high level languages to the C API, and as a nice side effect, the C-API also becomes safer and more "correct", because pointers and associated sizes can be passed around as one item.

(2) Make the bindings-generator configurable on a per-language and per-API basis, this can be as simple as a map which overrides type- and function-names, or injects manually written code into the generated bindings.

The goal is to make the generated bindings more idiomatic to the target language.

This mostly works if you have control over the underlying C-API of course, e.g. the language bindings are created by the original C-library project, not as an external project to convert a fixed C-API.

I wrote a blog post about this whole topic:

https://floooh.github.io/2020/08/23/sokol-bindgen.html

...and here's an example of such semi-auto-generated Zig bindings:

https://github.com/floooh/sokol-zig/tree/master/src/sokol

...for instance note the "injected" helper functions here:

https://github.com/floooh/sokol-zig/blob/1c93f60ad178869b84d...

arka2147483647• 5 years ago

I believe that it is best for this kind of automatic binding to expose the raw native types, and not use the language idiomatic ones.

I mean, if you need to use the raw win32 api, there must be a reason you need raw access, and in such a case, conversion to language types is more likely to be a hindrance than help.

Also, if you need to call multiple native apis, it is easier to pass the native types around if the is no conversion to language types.

pjmlp7• 5 years ago

The win32metadata repository has documentation on type renaming, and mapping integers into .NET enumerations.

It is up to the generation tool to provide idiomatic bindings based on the information provided by win32metadata.

This is nothing new, WinRT works exactly like this across .NET, C++, JS, Rust and whatever language projection one feels like adding.

steveklabnik• 5 years ago

What is new is the definitions for more than just WinRT.

Kaknut• 5 years ago

Oh so it arrived late, but better come now than never. Also it's super nice that I can use RUST for it.

smasher164• 5 years ago

Now if MacOS followed suit, all would be right in the universe.

benibela• 5 years ago

The best way to access Win32 APIs has always been Delphi's VCL

ed_blackburn• 5 years ago

Reminds me of Dan Appleman’s VB Win32 API book from ~20 years ago.

BelenusMordred2• 5 years ago

It's a great move but nearly every lang on Earth has a solid 3rd party Win32 library built over years and battle hardened.

Waiting patiently for Microsoft to actually put some effort into the obscure TPM documentation along with making it easier for developers to implement.

tinus_hn• 5 years ago

The TPM infrastructure is full of leaks that can’t be fixed, they’re not going to expose that with documentation. It’s like asking them to document a vendor to open-source their DRM.

BelenusMordred2• 5 years ago

full of leaks that can’t be fixed

Not claiming it's perfect, saying it exists on nearly every cpu running Windows today. There's serious value for plenty of people out there to have access to these interfaces that are only internally documented and available to a small group of people from one company alone.

Only have to look at the history of adding randomness to CryptographyAPI-NextGen to understand where I'm coming from.

No one has been able to add an additional RNG source to the Windows entropy pool for ~15 years now, highly venerable Microsoft developers say this is a good thing in public for all their enterprise clients when basic maths proves it wrong in about 2 minutes.

They do not have your best interests at heart, no matter how big you are, in fact it's less likely the bigger you are.

GordonS• 5 years ago

I mainly use C#, and this is late, but really welcome!

I usually use Vanara for Win32 PInvoke wrappera, or hand-code them if they are undocumented or I only need a couple. Rarely I've come across memory issues with the wrappers provided by Vanara and PInvoke.net, where they've forgotten to return a buffer - these kind of problems are absolute bastards to track down, and the root cause is of course manually building wrapper code. Having an automated means to do this will be a huge boon.

I wonder though - quite often if I'm digging into Win32 stuff, it means I'm working on performance sensitive code, where I want to keep allocations low - I haven't looked I to it yet, but I hope C#/Win32 has an option to generate wrappers that take pointers instead of only using managed types!

ziml77• 5 years ago

I gave their Rust bindings a try, but they still have work to do to even match the manual work that has been put into the winapi crate. I can say from experience is long, tedious, and error prone so it will definitely be better to be auto-generating the bindings.

dblock• 5 years ago

I was the de facto maintainer of JNA (https://github.com/java-native-access/jna) win32 bindings for years. The API metadata is welcome - we have always based all work on header files in the visual studio / windows SDK. Now there’s a change to generate that code - someone might want to try to replace hand crafted mappings in JNA with generated ones - there’s a huge test suite to work with.

pjmlp7• 5 years ago

First of all, thanks for the hard work.

I guess there is also the possibility to use the JNI replacement when it comes out of preview, although that would be constrained to Java 17 or later.

jedisct1• 5 years ago

Interesting. The real thing here is the Win32 Metadata project: https://github.com/microsoft/win32metadata

It contains definitions that can be used to automatically generate bindings to any language. So, we have C# and Rust as examples, but the very same definitions could be used with languages like Zig.

Very cool initiative.

Making Win32 APIs More Accessible to More Languages

Related Stories