SciPy builds for Python 3.12 on Windows are a minor miracle

https://labs.quansight.org/blog/building-scipy-with-flang

by todsacerdoti • 3 years ago

475 292 3 years ago

labs.quansight.org

pklausler2• 3 years ago

I'm very glad that f18 worked (so far as we know). I haven't tried building SciPy for any platform yet myself, much less Windows.

gamache2• 3 years ago

One nit to pick: as far as I am aware, "aarch64" and "arm64" are the same thing. Am I off?

nerdponx3• 3 years ago

What I see in Python is that "aarch64" usually refers to Linux and "arm64" usually refers to MacOS ARM. I don't know enough about these things to understand why they have different names.

mschuster91• 3 years ago

They are the same thing but there used to be two competing LLVM implementations on the backend side.

[1] https://www.phoronix.com/news/MTY5ODk

gamache2• 3 years ago

Thanks, that is very helpful!

stefan_• 3 years ago

Obligatory: https://lkml.org/lkml/2012/7/15/133

pitaj• 3 years ago

I love how he ranted so hard - without even mentioning the embedded or realtime variants.

cozzyd2• 3 years ago

the build system churn in python is really hard to keep up with.

I'm also curious about performance numbers on Windows (though, to first order, it doesn't matter... anything serious is probably running on a Linux machine).

nerdponx3• 3 years ago

Fortunately it should be slowing down now. The big transition was getting everyone on board with PEP 517, especially converting legacy Setuptools projects.

lmm2• 3 years ago

I've been hearing something like this about Python builds every two or three years for decades. How long until the next "big transition" is "converting legacy PEP 517 projects"?

hurryer5• 3 years ago

For pure CPU computation Windows is just as fast as Linux since 99.9% of the time it's your code running and not the OS.

murderfs2• 3 years ago

Depending on the code you're running, calling convention can matter a lot. The SysV ABI will use xmm/ymm/zmm automatically, whereas on Windows you have to opt into it with __vectorcall.

sdfhioandion• 3 years ago

Not true. Changes to the OS, particularly to the scheduler, can affect CPU-bound work a great deal. It's "your code", but the OS decides when and where it runs. For example, the changes between Linux 6.5 and Linux 6.6 led to a >20% uplift in TensorFlow and some smaller uplifts to Blender. This is the same software on the same hardware.

https://www.phoronix.com/review/linux66-epyc-xeon/3

I couldn't track down a more detailed scheduler-specific benchmark. I remember reading one on Phoronix a few months back...

cozzyd2• 3 years ago

Not many compute clusters run windows

drtournier• 3 years ago

https://xkcd.com/2347/

josh-sematic• 3 years ago

Also https://xkcd.com/1987/

Edit: not to imply that the work of the maintainers hasn’t been INCREDIBLE (it has). I just thought this XKCD was a funny take on how complex the python packaging ecosystem is.

tedunangst2• 3 years ago

In particular, Meson was going to refuse to accept the MSVC+gfortran combination that was in use in conda-forge.

This sounds like a bug? The point of a build tool is to run the commands you tell it, not tell you, sorry dave.

rbancroft• 3 years ago

The article is well-written and detailed, however I was taken aback by the claim that meson is 'widely used for C & C++ projects'. I've come across bazel more often than meson. I guess because meson is written in python it seemed like a good choice for SciPy, and it worked out in the end so congratulations.

But yeah, I think CMake is still the gold standard despite all its quirks, complexities and problems.

bsder3• 3 years ago

Meson is a front end.

Edit: Caution: This statement is incorrect--It can generate CMakefiles as a backend. It can also generate build files for MSVC. It can also generate standard Makefiles.

Edit: Corrected downstream. CMake can make Ninja files which Meson also makes. I got this backwards but kept the edit so people won't get confused.

CMake is NOT a gold standard. CMake is an agglomerative disaster.

For example: try getting CMake to accept zig as your compiler. "Oh, your compiler command has a space in it? So sorry. I'm going to put everything after the space in all manner of weird places. Some correct--some broken--some totally random." If you're lucky CMake crashes with an inscrutable message. If you're unlucky, you wind up with a compiler command that fails in bizarre ways and no way to figure out why CMake is doing what it did.

This is my experience with CMake every damn time--some absolutely inscrutable bug pops up until I figure out how to route around it. If I'm really unlucky, I have to file a bug report with CMake as I can't route around it.

Sure, if some unfortunate soul has beaten CMake into submission and produced a functioning CMakefile, CMake works. If YOU are the poor slob having to create that CMakefile, you are in for worlds and worlds and worlds of pain.

wenc3• 3 years ago

Projects that use Meson

https://mesonbuild.com/Users.html

Includes Gimp, Gtk+, nautilus, Postgres, qemu, Wayland…

gavinhoward4• 3 years ago

I agree.

Disclaimer: author of a soon-to-be released Meson competitor.

But one obscure, little gem of a rant that taught me what you just said is [1].

tl;dr: Build systems should run the commands you tell them to run, period. Because sometimes, the programmer actually does know what he's doing.

I am ashamed to admit that before I read that comment, I was thinking about making my build system magical. But after reading that comment, I realized that "magic" is why people hate build systems.

[1] https://ofekshilon.com/2016/08/30/cmake-rants/#comment-29273

chubot2• 3 years ago

I read this 2021 comment/rant and didn't find it illuminating or insightful

I just want a tool that doesn’t make ANY assumptions about compiler flags or compilers or how my project directory structure is laid out. I can do the leg work of inputting all the exact parameters and build configurations into the tool. I just need the tool to incrementally compile my code in a parallel fashion without HACKS

That's exactly what Ninja is, and it's existed since 2012.

CMake is a flawed generator for Ninja, but you can write your own. I did that for my project, as explained here - https://lobste.rs/s/qnb7xt/ninja_is_enough_build_system#c_tu...

gpderetta• 3 years ago

Magic is great when you write the magic to suit your problem.

When you are subject to other people magic, it usually ends in tears.

aidenn0• 3 years ago

Meson does more than run the command you tell it, it can also synthesize those commands (so you can support MSVC/gcc/clang without writing build rules). If you ask it to synthesize a command for a combination it doesn't know about, of course it's going to tell you "sorry dave"

tedunangst2• 3 years ago

Yeah, sure, 40 years ago make had implicit rules for building .o files from .c files so you didn't have to write them, but if you told it your compiler was going to be bananac, it wouldn't refuse to run.

bonzini3• 3 years ago

It's the MSVC linker that would complain. The problem is that the C runtimes used by MSVC and gfortran (whose own runtime library is written in C) are not ABI-compatible. The hack that numpy used was to link the Fortran objects into a DLL to add a level of indirection (the import library) that would pacify MSVC.

So there was some extra work needed to create these DLLs. Either in the build description files, or in Meson. The SciPy people didn't want to implement this indirection in either place, and the Meson developers were not eager to help them either (they did help in general, for example with Fortran and Cython support; but they don't want to provide footguns) because it was indeed a hack. It only worked because the Fortran side didn't use files that were opened on the Python/C side, for example.

https://web.archive.org/web/20180711144501/https://pav.iki.f...

bsder3• 3 years ago

Thank you for the clarification. Meson refusing to do something and the developers not deeming that a bug seemed really unusual.

wenc3• 3 years ago

What an amazing read. Now I know why my pip installs are failing in 3.12 but we now have a brighter future ahead.

Also while I love Python it’s helpful to understand why Python packaging is a (manageable) mess. It’s because of non standardization of build tools for C/C++/fortran and the immensity of the ecosystem, nothing to do with Python itself. It’s part irreducible complexity.

It’s a miracle it works at all.

IshKebab5• 3 years ago

It's also to do with Python itself.

wenc3• 3 years ago

How?

cdavid2• 3 years ago

Yes, that's a fundamental reason python packaging is a mess. Python success is largely due to the availability of key mixed language packages. No other mainstream language package manager has to deal with this.

For example, cargo for rust, which is great, can assume to package mostly rust-only code. And while it is compiled, the language "owns" the compiler, which means building from sources as distribution strategy works. I don't know how/if cargo can deal with e.g. fortran out of the box, but I doubt cargo on windows would work well if top cargo packages required fortran code.

The single biggest improvement for python ecosystem was the standardisation of a binary package format, wheel. It is only then that the whole scientific python ecosystem started to thrive on windows. But binary compatibility is a huge PITA, especially across languages and CPUs.

NewJazz• 3 years ago

Doesn't R handle fortran and C++ packages?

winstonewert3• 3 years ago

Many rust crates actually do package code written in other languages. There are plenty of useful C/C++/Fortran libraries that nobody has rewritten into Rust but for which wrappers have been created that call into C. It works in Rust because the build.rs lets libraries do whatever they want during the build process, including invoking compilers for other languages.

Various factors still make the Rust and Python story different (Python uses more mixed-language packages, the Rust demographic is more technically advanced, etc). But a big one is that in Rust, the FFI is defined in Rust. Recompiling just the Rust code gives you an updated FFI compatible with your version of Rust. In Python, the FFI is typically defined in C, so recompiling the python won't get you a compatible FFI. If Python did all FFI through something like ctypes, it would be much more smooth.

the__alchemist• 3 years ago

As you might expect, compiling rust crates that use C libraries can lead to the inscrutable-block-of-text linker errors we know and love. I have been having a rough time with CUDA, CMSIS-DSP, Tensorflow, and OpenCV during the past few weeks. One of them requires LLVM=v15 to be installed; another requires v16+. A diff one requires an old version of rustc to be installed. On a diff one, when I posted on Github, the maintainers assured me the crate is, in fact, fine, and my system is misconfigured; phew!

jchw• 3 years ago

I don't think this is really true. NodeJS has quite a similar problem.

shoo• 3 years ago

The single biggest improvement for python ecosystem was the standardisation of a binary package format, wheel.

I agree. Some people love to complain about python packaging. But from one perspective, it's arguably been a solved problem since wheels were introduced 10 years ago. The introduction of wheels was a massive step forward. Only depend on wheel archives, don't depend on packages that need to be built from source, and drag in all manner of exciting compile-time dependencies.

If there's a package you want to depend on for your target platform, and the maintainers don't produce a prebuilt wheel archive for your platform -- well, set up a build server and build some wheels yourself, and host them somewhere, or pick a different platform.

sertbdfgbnfgsd2• 3 years ago

Yes, that's a fundamental reason python packaging is a mess. Python success is largely due to the availability of key mixed language packages. No other mainstream language package manager has to deal with this.

Admittedly I'm not a python expert, but julia handles this just fine? It doesn't seem like it's a difficulty inherent to "mixed language packages". Somehow it appears to me that python's approach is just bad somehow.

smabie• 3 years ago

Nothing to do with Python? These FFI bindings exist because Python is slow as dirt.

zo1• 3 years ago

And the bindings to python exist because those languages are a pain in the ass to work with.

I don't want to work with Fortran, C++, Cobol, etc. And I sure as hell don't want to figure out how to integrate such wildly different languages into my existing and modern ecosystem.

baq7• 3 years ago

These FFI bindings exist because Python was designed as a glue language for FFI bindings.

zzbn00• 3 years ago

This is one of the reasons why Python got so popular.

It is too slow to reimplement big pieces of software in it, so people just used bindings to existing code. And productivity rocketed!

globalnode• 3 years ago

yes it was very eye opening. i often see people comparing their favourite package manager to pythons and coming to the conclusion that python is terrible, but its not! one thing i dont quite understand, is why dont the python people just use a c/c++ math library instead of fortran?

patrick451• 3 years ago

There often simply doesn't exist an equivalent library written in c/c++ (or any other language for that matter). The example I'm familiar with is SLICOT (Subroutine Library in Systems and Control Theory) [1], exposed in python through Slycot [2]. It as routines for pole placement, riccati solvers, various factorizations, and MIMO zero computations and a ton of other stuff. As far as I have been able to find, no c/c++/other-language comes close to supplying either the breadth or depth of this library. Further, many of the SLICOT subroutines were written by the original inventors of their respective algorithms, which I view as a big bonus.

[1] http://slicot.org/ [2] https://github.com/python-control/Slycot

ecshafer2• 3 years ago

Imo i would much rather write a math library in fortran than c or cpp. Fortran is quite a joy for doing numerical work. It sucks for most other things though. Really the only thing nowadays is that youll probably be using gpus so that makes cpp better for cuda integration.

dzdt• 3 years ago

Because some of the best math libraries are written in Fortran. Seriously lots of heavy duty scientific code was written in Fortran in the 1970s and is still underpinning applications today. In many cases there is no equivalent alternative.

keithalewis• 3 years ago

Indeed. The real problem is python seems to attract people with no training in software development. It is a mess on top of a mess.

dylan604• 3 years ago

I thought that was what people said about php

tomlockwood• 3 years ago

That's also a feature - by design it has to be friendly to new users, and not an arcane art only accessible to the Chosen One, as much as those Chosen Ones would like to be the only programmers.

placebo• 3 years ago

It’s a miracle it works at all.

I agree. In fact with what seems to be an exponential growth in complexity of software ecosystems, what's keeping it all from eventually getting to a "tower of Babel" catastrophe? Of course, this does not only apply to software, but it is a good example.

bee_rider2• 3 years ago

I’m under the impression that the best BLASes are mostly C (MKL, Blis, and OpenBlas). I wonder how far they could get with just C and Python.

I wonder if they’d just go with libflame instead, if they started today.

Of course there’s lots of other functionality in scipy; iterative stuff, sparse stuff, etc etc, so maybe Fortran is unavoidable (although, Fortran is a great language, I’m glad the tooling situation is at least starting to improve on Windows).

cdavid2• 3 years ago

The question of removing fortran from scipy has happened a few times, and never got anywhere for the reasons you gave. A lot of scipy itself contains fortran code that would take man years to rewrite.

Several key parts using fortran have been removed, once it became possible. For example fft stuff.

pharrington• 3 years ago

not gonna lie, seeing "Fortran is a great language, I’m glad the tooling situation is at least starting to improve on Windows" was not on my 2023 bingo card

bee_rider2• 3 years ago

I assume that is because Fortran is such a great language that you just didn’t think it was possible that the tooling situation on Windows could be anything but perfect. :)

d3m0t3p• 3 years ago

In their table about compilers, isn't AArch64 and ARM64 the same ?

tempay• 3 years ago

Yes, though different operating systems use different naming. On Linux it's `aarch64` and on macOS+Windows it's `arm64`.

Grimburger2• 3 years ago

GCC..there are ways to make it work on Windows (through cygwin or MinGW)

I assume it would be working on WSL also? Which seems like a simple solution to the mess that is developing on Windows, especially in Python.

mkoubaa3• 3 years ago

No, Python on windows needs to work natively without wsl.

Grimburger2• 3 years ago

I'm struggling to comprehend your comment here, python needs to work native on windows or is it something else, why is WSL such a barrier given it's generally available on a few billion devices worldwide?

FrustratedMonky2• 3 years ago

Maybe move away from Python altogether?

mkoubaa3• 3 years ago

Great idea! Why don't you go suggest to the developers of sciPY??

FrustratedMonky2• 3 years ago

The developers of SciPY are really good, maybe their brains could be focused on developing tools for a different language that doesn't cause articles to be written about how crazy it is to integrate with it.

At some point, move on, Python started as a toy, and grew into a behemoth.

nologic01• 3 years ago

Python is now bigger than what even its most keen friends ever imagined might happen. This is not automatically solving the issues, nor make it more likeable to those who dislike it, but it does incentivise serious plumbing work.

dilawar• 3 years ago

SOS Justine's Cosmopolitan.

mattbillenstein• 3 years ago

Very naive question, but are the semantics of Fortran so different that it can't be translated to C first and then compiled using a C compiler? Perhaps maintained in C going forward?

I can't imagine there are a lot of Fortran folks around maintaining these old libraries - they must need maintenance no?

googl-free• 3 years ago

yeah, fortran has native arrays

lldb• 3 years ago

We have this already - “f2c” has been around for decades.

ogrisel• 3 years ago

It's currently used to provide a hackish WASM compatible port of scipy for Pyodide and the Pyodide maintainers are eager to be able to drop the f2c hacks in favor of lfortran or any WASM-capable fotran compiler because f2c can cause very hard to debug low level crashes when running scipy or other downstream libraries tests (e.g. scikit-learn).

mkoubaa3• 3 years ago

Fortran is a higher level language than C. There are plenty of fortran developers.

It's awful for application development, but that isn't it's niche

not2b2• 3 years ago

If you want the code to run slower, yes, you could do that. Because there are no pointers in Fortran, only arrays, and because arguments to functions aren't allowed to alias (let's ignore the horror of COMMON blocks for now), aggressive optimization and vectorization is easier.

The standard Fortran math libraries just work, and they are fast.

I should clarify that you can write C/C++ code that would have equivalent speed, especially with the C restrict keyword, but putting in an f2c step to translate the existing code will making things significantly worse in many cases.

Sohcahtoa823• 3 years ago

Because there are no pointers in Fortran, only arrays, and because arguments to functions aren't allowed to alias [...], aggressive optimization and vectorization is easier.

Can you explain this like I'm 12?

What's an example of an aggressive optimization you can make based on arguments not being aliased and there being no pointers?

dwheeler• 3 years ago

Also, in Fortran arrays are normally stored in column-major order (sometimes called “Fortran order”). C uses row order. A simplistic translation would have terrible performance.

pklausler2• 3 years ago

Fortran has had pointers in the standard language since F'90 and as a ubiquitous vendor extension since ca. 15 years earlier.

tsegratis• 3 years ago

The κατα in καταστροφή means 'down to' or 'according to' rather than sudden, and implies strongly a bad turn

The inverse of κατα is often ανα though αναστροφή means literally "up turn" or invert

So maybe ευστροφη a "good turn" (eustrophe) would be better coinage

But arguing with JRR Tolkien on language coinage and being right would be a ... ευκαταστροφη i.e. good luck!

But in general I love their noticing of the overflowing grace -- that's something that gives joy and happiness in the world

coldtea• 3 years ago

"Kata" in katastrofi actually means "against", so katastrofi is things "turning against".

richwater• 3 years ago

Yet another reason the Python ecosystem and installation is unfixable.

LispSporks22• 3 years ago

I have thought that everyone was using WSL2 for this kind of thing and calling it a day. What are the reasons for even trying to build a native Windows version?

PLenz• 3 years ago

Way more researchers use windows then you might think. Plus students.

spookie5• 3 years ago

Maybe I'm biased, being in EU and all. But I've seen more Linux than Windows in those environments.

RockRobotRock• 3 years ago

msft and nvidia tackling CUDA drivers on WSL2 is a godsend. Especially when you can use Docker Desktop on it.

Definitely making the best of a less than ideal situation.

hurryer5• 3 years ago

That's like asking way don't mac developers work in a linux vm instead of wanting a native build.

Because working in a VM is inconvenient and has poor integration.

satvikpendem4• 3 years ago

WSL2 is not really a VM though, in the traditional sense, as it has full hardware access, including the GPU. Technically, when you enable WSL2, the Windows build itself is running through the Hyper-V hypervisor, as is the Linux distribution. In fact, a few years ago, I used Proxmox to basically do the same thing, in order to test cross-desktop apps I was developing for Windows, Linux, and macOS (as a Hackintosh).

ActorNightly10• 3 years ago

WSL2 is both convenient and is extremely well integrated. Microsoft did a really good job with it and VSCode.

Also, in most cases for scientific computing, Mac users are using stuff like Remote SSH in VSCode to work directly on hardware that is running the code, which is pretty much always Linux.

Generally, its kinda said that manpower is being wasted on getting things running in Windows or Mac by the software developers. It should be the other way around, have Microsoft or Apple dedicate manpower to port things over and make both systems conform to the Linux standard.

nijave• 3 years ago

I'm guessing big corporations where it's near impossible to get a non-Windows machine are a big part of the audience.

aj7• 3 years ago

Does anyone else find the handling of arrays by Python so horrific that they can’t bring themselves to use it?

kstrauser• 3 years ago

In what way?

__MatrixMan__• 3 years ago

Do you mean lists, or is there some array type that I'm not thinking of?

hurryer5• 3 years ago

There is the array module, but it's cumbersome to use.

https://docs.python.org/3/library/array.html

But unlikely this is what parent meant.

ketozhang• 3 years ago

It's not popular because you're mostly hearing from the science community who want more features in their array (vector/matrix/tensors).

Why would you want to use C-like arrays in Python anyways?

avidphantasm2• 3 years ago

Great read. After spending a lot of time this year modernizing a CMake C++ project with Python bindings, which I successfully added to conda-forge as a new feedstock, I can say with great confidence that the first IT-related change I would make as God Emperor would be to extirpate Windows from all Universes for Eternity.

panzi• 3 years ago

About the compiler/architecture table: Whats the difference between arm64 and aarch64? The Wikipedia article to ARM64 redirects to AArch64.

stingraycharles• 3 years ago

I was thinking the same thing. Maybe they meant the armhfp architecture?

pietroppeter• 3 years ago

> While Fortran has long been the butt of the joke in IT departments the world over, in a curious twist of fate, it has seen a dramatic resurgence over the last few years. While the reasons for this are not exactly obvious ...
It is obvious to me. :)

https://x.com/OndrejCertik/status/1722364038212899274?s=20

resurrecting fortran: https://ondrejcertik.com/blog/2021/03/resurrecting-fortran/

bjornasm• 3 years ago

This was a great read, thx :) Of all the other languages than Python, I am the most drawn to Fortran. I used it for some university project and it was quite nice. Hope I'll find a good excuse to pick it up for some project.

kragen14• 3 years ago

maybe the free software community should stop subsidizing microsoft's operating systems and let them port things like scipy to it themselves

after all, if you want linux, you know where to find it; it's right there in wsl2

also microsoft could start shipping a fucking compiler with their sorry malware-ridden excuse for an operating system, like every single other operating system vendor has for sixty years

xcv1236• 3 years ago

microsoft could start shipping a fucking compiler

Visual Studio is a free download. Most users are not developers. Waste of space to include it by default.

https://visualstudio.microsoft.com/vs/community/

kragen14• 3 years ago

that's a ridiculous excuse

unix v7 included a compiler and was three megabytes

https://www.tuhs.org/Archive/Distributions/Research/Keith_Bo...

https://opensimh.org/research-unix-7-pdp11-45-v2.0.pdf

the compiler was a tiny fraction of that

gcc 9 is about 50 megabytes

windows 11 is over 8 gigabytes; you need a 16 gig usb drive to install it

there are probably individual audio files included in windows that are bigger than gcc

also, tho, this is a lot like not including life jackets on a ship because most passengers don't get shipwrecked

gary_02• 3 years ago

All you really need is cl.exe and friends, and the C/C++ headers, in some fixed directory path. (And, apparently, maybe a Fortran compiler?) That shouldn't take up that much space.

And the last time I was forced to install Windows 10, it spent a few hundred megabytes of bandwidth and disk space on Candy Crush Soda Saga, not to mention a bunch of other junk I never asked for, so disk space is not that precious to Microsoft.

warangal• 3 years ago

Indeed it was a great read about what goes behind the scenes for maintaining such packages and a problem generally every developer thinks about when such an extension has to be shipped along .

I still wonder if targeting Python's LIMITED C API wouldn't help in this case. I use a tool (for Nim) which seems to target that Limited API and solves the problem of specific python version mismatching at-least for my specific code. I never had to upgrade a pure C/Nim extension due to python version !

Using zig-cc along with a fixed GLIBC (2.27) and generic CPU flag also made it possible to target linux ecosystem in case user is not a developer and just use the compiled extension shipped with the package.

eviks9• 3 years ago

Heroes and miracles are indeed required in the swamps of bad designs

vintagedave2• 3 years ago

An amazing read. Most of the article notes that there was no Fortran compiler yet (though amazing how well the one they landed on worked -- a miracle alright!) but I was particularly struck by the ABI issues. The story of a lone developer hacking a MinGW build that used the right ABI struck home.

I can provide some background on the ABI issues in Windows.

The following is personal opinion not company opinion, and I am a product manager not an engineer so may have some technical details wrong. But:

At Embarcadero we're moving our C++Builder toolchain forward to a new Clang[1], and using a new C and C++ RTL layer [also see 1]. Like SciPy, we're now using UCRT, and the key bits that cause difficulty are the C++ RTL and platform ABI. Boy howdy do we have some stories.

Issues:

* There is no standard Windows ABI beyond what msvc produces. This results in WinAPI (C-level, in other words!) APIs that could not in the past be reliably used from anything other than msvc because they could throw exceptions, rather than returning error codes, and Win64 SEH is not fully documented (clang-cl, which is open source, disappears into the closed source msvcrt to handle this.)

* Or another issue: cdecl isn't standardised. This may surprise those of you who think -- as I used to think! -- that cdecl is simple and a known calling convention for any platform. We have issues where sret for return values[2] can be different between our toolchain and something built with MSVC. Since we need to change multiple languages, and one of our languages (Delphi) handles some returns (managed types) differently, changing this is more complex than it looks. We already did a lot of ABI compatibility work several years ago between multiple languages[3].

Back to the ABI: our new Clang is aiming at being fairly compatible with mingw-llvm on Windows. (And MSVC too, but we're starting with mingw-llvm as the basis.) That does not include a C++ ABI which the C++ committee has been resistant to, but it's a known good, working C++ toolchain, open source.

If Windows ever did have a platform ABI, it would likely be based on MSVC, but I would suggest that we -- developers -- should resist that until or unless the entire toolchain including runtime internals that affect the ABI is open sourced, or at least documented so that other toolchains can match it.

[1] https://blogs.embarcadero.com/win64-clang-toolchains-in-rad-...

[2] sret is a hidden (?) or special return value used for structs, basically when a large (> register size) memory is required. I think. This is where I hope I don't embarass myself too much in the explanation. See eg https://stackoverflow.com/questions/66894013/when-calling-ff... In other words, returning values is more complex than it seems, even for a plain simple C method returning a struct which should be _incredibly basic_ and becomes a complex ABI issue.

[3] https://blogs.embarcadero.com/abi-changes-in-rad-studio-10-3...

dist-epoch• 3 years ago

closed source msvcrt

Isn't msvcrt source code included with Visual Studio? I distinctly remember looking through it many years ago.

And I see it now too on my install: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\crt\src

But maybe it's not 100% of it and some key parts and missing, and obviously there might be legal issues reading this source code.

ack_complete• 3 years ago

Not complete. Some parts are licensed, such as math libraries, and VS dropped the ability to rebuild the CRT DLL externally some time ago. A lot of it does have viewable source, but some critical core routines are only shipped compiled.

mike_hearn2• 3 years ago

Sometimes when a detail of Windows isn't documented, the Wine source code can be useful. Have you tried looking at it for details of win64 SEH? For example:

https://github.com/wine-mirror/wine/blob/master/dlls/ntdll/e...

https://github.com/wine-mirror/wine/blob/master/dlls/msvcrt/...

vintagedave2• 3 years ago

We have not, simply for legal reasons. The Wine folk are wizards, though.

bjackman• 3 years ago

Back when Linux was a janky corner case run by loosely-coordinated hackers with sometimes-impractical ideological constraints, when brilliant people were expending heaps of effort to enable support for it, that was fantastic.

Now that the janky corner case is a proprietary system run by cyber-landlords whose constraints are just... hostility, I feel less positive about the work being done to support it.

On the one hand it's really great that these people care so deeply about making these tools usable for everyone. And I applaud them for doing it, absolutely not suggesting they should change course, just pondering.

But... now instead of thinking "wow I'm so glad this work is happening" I do rather think "wow imagine what those wonderful people could achieve if they didn't have to work on this".

SiempreViernes• 3 years ago

Well, "they don't "have" to work on it, as was pointed out several times the SciPy devs are volunteers.

Indeed most of the story is explaining why SciPy just had to hope someone else would make a open source Fortran compiler for windows, and it looks like it was mainly NVidia devs that provided salvation

henrydark• 3 years ago

I must go on compiling / You can't break that which isn't yours / I must go on packaging / I'm not my own, it's not my choice

crabbone5• 3 years ago

It's not even that. SciPy is paying the price of the idiotic (and highly biased) decisions of Python core-dev members who chose MSVC over MinGW for Python on Windows. (And their motivation is derived from Microsoft sponsorship. A bunch of people on core-dev list are straight-up Microsoft employees, who are paid by Microsoft to be on that list, and, on top of that, Microsoft pays for CI servers for CPython project).

This whole problem could've been avoided if Python just didn't use proprietary tools in its toolchain.

IcyWindows• 3 years ago

Are you saying using llvm would have also made things better, or are you saying that it would be better if everyone just used gcc?

Decabytes• 3 years ago

Sometimes I wonder if Rust’s biggest draw is not that is is safe, but that is removed a lot of the BS hoops you have to jump through to get a working program on your computer.

I wonder if the Python alternatives will also get a similar boost for that reason. If maintaining the language and the ecosystem is that much of a bear, it would save a lot of human hours to do that. Even if we started back from scratch for a bit

ngcc_hk• 3 years ago

Just go to read: https://www.bitecode.dev/p/relieving-your-python-packaging-p...

TL; DR

The reason why we need virtual this and that in python is that some work in one and some work in the others. This is particularly problematic in macOS where sometimes the system fall back to the default (somewhat default) python compiler. I even have script to check version and lots of environment for running different program.

It is a mess.

After this magic, is there a new packaging environment, procedure, ... for the rest of us?

crabbone5• 3 years ago

I just want to comment on this:

Meson was going to refuse to accept the MSVC+gfortran combination

Back in the days, I went to Python's core-dev list and asked why. Why would any sane person ever use MSVC for a cross-platform language runtime. And guess what the answer was? Well... The answer was "Microsoft pays us, gives us servers to run CI on, and that's why we will use Microsoft's tools, goodby!"

For reference, Ruby uses GCC for the same purpose as do plenty of other similar languages for this exact reason.

To give you some context, I ran into this problem when writing bindings to kubectl. For those of you that don't know, in order to interface with Python from Go, one needs CGO, and on MS Windows it means MinGW. You could, in principle, build Python itself with GCC (a.k.a. MinGW) (and that's what MSYS2 a.k.a Cygwin a.k.a. Gitbash does), but this means no ABI compatibility with the garbage distributed from python.org.

So, after I had a proof of concept bindings to kubectl working on Linux, I learned that there will be no way (well, no reasonably simple way) to get that working on Windows. So, the project died. (Btw, there still isn't a good Kubernetes client in Python).

---

On the subject of packaging. I've decided to write my own Wheel packager. Just as a way to learn Ada. This made me read through the "spec" of this format while paying a lot more attention that I ever needed before. And what a dumpster fire this format is... It's insane that this atrocity is used by millions, and so much of critical infrastructure relies on this insanity to function.

It's very sad that these things are only ever discussed by a very small, very biased, and not very smart group of people. But then their decisions affect so many w/o even the baseline knowledge of the decisions made by those few. I feel like Python users should be picking up pitchforks and torches and marching on PyPA (home-)offices and demand change. Alas, most those adversely affected by their work have no idea PyPA exists, forget the details of their work.

mxz3000• 3 years ago

hindsight is 20/20.

Don't criticise people for making certain decisions years ago when those don't match what you'd choose to do now. Often you'll find that they were very reasonable given the constraints at the time.

Also the spec will have evolved over time with changes that would have been made under constraint of the existing system, which tends to produce things that are not as nice compared to something that was designed from the get-go to support the features. This is something that's seen very often in software engineering, and are probably partly a reason why long-lived codebases tend to be dumpster fires in general.

Calling them 'very biased and not very smart' is not very constructive.

That's not to say that the wheel format isn't a dumpster fire (I'll have to take your word on that), or hasn't morphed into one with time & revisions.

crabbone5• 3 years ago

Are we talking about not criticizing Wheel format?

Because, if so, I'm not buying it. Wheel is an iteration after Egg, that was created in a world full of package managers, packages of all sorts and flavors. Wheel authors failed to learn from what was available for... idk some odd thirty years? (I'm thinking CPAN).

But, it has problems that just show how immature the people who designed the format were when it comes to using existing formats. For example, the Wheel authors were completely clueless about multiple gotchas of Zip format (even though they've been using Egg which is also based on Zip for... what a decade? I mean, come on, you had to be blind and deaf not to know about these problems if you had anything to do with packaging).

But, the most important problem is in the name format. And it's not about knowing gotchas of other formats. It's just total lack of planning / ability to predict the next step. For instance, some parts of the Wheel name are defined roughly as "whatever some function in sys module returns on that platform". So, it leaves this part of the name unpredictable and undefined, essentially. Wheel authors cannot make a universal package because in order to do so they need to have knowledge of all existing platforms and all future platforms... which, of course, nobody does.

And they've done it because... it was easy to do. Not because it was the right thing to do or the smart thing to do. The consequence of this decision is that implementing a PyPI competitor is virtually impossible because it's a layered crap-sandwich of multiple layers of mistakes that support each other (various parts of the name format were modified multiple times over the course of history, and weren't immediately supported by pip). Similarly, implementing a viable alternative to pip is equally almost impossible because of the same historical crap-pie of multiple mistakes on which Python package publishers built their whole infrastructure.

This led to the situation where today the whole Python packaging is locked into using PyPI, setuptools and pip. Those who are intimately familiar with the subject know that they are broken beyond repair and have no hope of getting better, but the mess is so big that undoing it just seems impossible. And, of course, PyPA is blissfully unaware of all the nonsense that's going on in its tools keeps adding new worthless features to polish this turd.

hazrmard• 3 years ago

https://archive.is/SyeRt

Archive link. Website doesn't load for me.

carapace• 3 years ago

Having finally gotten over the whole 2->3 fiasco I'm still sitting out new Python development until the PyPA sorts out the new packaging & distribution story. They're working on it and making progress, but it's still pretty gnarly.

Anyway, the complexity is too damn high!

swingingFlyFish• 3 years ago

This is a really insightful article. Thank you for sharing it. it reminds me there is still so much for to be done in the FOSS community and how grateful we are to have it. Thank you!

SciPy builds for Python 3.12 on Windows are a minor miracle

Related Stories