Cosmopolitan v3.5

https://github.com/jart/cosmopolitan/releases/tag/3.5.0

by sgammon • 2 years ago

366 143 2 years ago

github.com

WantonQuantum• 2 years ago

I assume the executables won't run on ARM platforms? So modern Macs are out, etc.

nabla9• 2 years ago

It works on AMD64 and ARM64 both.

It makes two architecture fat binary, but removes all code not needed. Resulting binary is typically only 30-50% larger than 1-architecture binary.

mkl2• 2 years ago

More details of supported platforms at https://justine.lol/cosmo3/ and https://github.com/jart/cosmopolitan/blob/master/tool/cosmoc..., both linked in the release notes.

lukemerrick• 2 years ago

Looks like there is both an ARM and x86 version according to the docs. Probably need two different binaries, but you still get cross-OS for each architecture.

eesmith3• 2 years ago

It's a single binary, cross-OS, cross-arch.

Go to https://cosmo.zip/pub/cosmos/bin/ and download an executable, like https://cosmo.zip/pub/cosmos/bin/basename .

  % curl -O https://cosmo.zip/pub/cosmos/bin/basename
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  100  663k  100  663k    0     0   440k      0  0:00:01  0:00:01 --:--:--  441k
  % chmod +x basename

On macOS with an M1:

  % arch
  arm64
  % sha256sum basename
  2e4cf8378b679dd0c2cdcc043bb1f21eaf464d3c3b636ed101fbc5e3230eb5fb  basename
  % file ./basename
  ./basename: DOS/MBR boot sector; partition 1 : ID=0x7f, active, start-CHS (0x0,0,1), end-CHS (0x3ff,255,63), startsector 0, 4294967295 sectors
  % ./basename /dev/null.txt
  null.txt

On FreeBSD with an amd64:

  % uname -sm
  FreeBSD amd64
  % sha256sum basename
  2e4cf8378b679dd0c2cdcc043bb1f21eaf464d3c3b636ed101fbc5e3230eb5fb  basename
  % file basename
  basename: DOS/MBR boot sector; partition 1 : ID=0x7f, active, start-CHS (0x0,0,1), end-CHS (0x3ff,255,63), startsector 0, 4294967295 sectors
  % ./basename /dev/null.pdf
  null.pdf

d4rti• 2 years ago

Works for sure:

  $ file ./bin/python
  ./bin/python: DOS/MBR boot sector; partition 1 : ID=0x7f, active, start-CHS 
  (0x0,0,1), end-CHS (0x3ff,255,63), startsector 0, 4294967295 sectors

  $ ./bin/python
  Python 3.11.4 (heads/pypack1:6eea485, Jan 24 2024, 10:14:24) [GCC 11.2.0] on 
  linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import platform
  >>> platform.processor()
  'arm'

4gotunameagain2• 2 years ago

Out of curiosity, why are you running python packed with cosmopolitan ?

smcameron• 2 years ago

I forget how I did this, but:

    $ file hello.com
    hello.com: αcτµαlly pδrταblε εxεcµταblε

Aissen2• 2 years ago

It depends. It does not work on CentOS 8 / Rocky 8 for example: https://github.com/jart/cosmopolitan/issues/1151

jart9• 2 years ago

Is that an embedded device with a small address space? APE used to require a 47bit address space. The release I published today fixes that. Maybe give it another try?

speedgoose• 2 years ago

The repository also contains the source code for redbean, which is a neat web server with lua and plenty of features.

https://redbean.dev/

https://github.com/jart/cosmopolitan/tree/master/tool/net

sgt_bilko• 2 years ago

The Lua fullmoon web framework is built around redbean. https://github.com/pkulchenko/fullmoon

morbicer• 2 years ago

Wow this is awesome, thanks for the tip

ncruces• 2 years ago

Having written a portable SQLite VFS from scratch, I'm a bit scared of how they blindly translate fcntl locks into LockFileEx in a forking web server.

The opportunity for data corruption just freaks me out.

m1keil• 2 years ago

Cosmo looks so cool. Are there any big projects out there that use it for distribution?

PhilippGille• 2 years ago

Llamafile: https://github.com/Mozilla-Ocho/llamafile

mkl2• 2 years ago

Related re Windows on ARM https://news.ycombinator.com/item?id=40772475.

trustno2• 2 years ago

I'm always somewhat scared - not sure if rationally or irrationally - about cosmopolitan.

It's a cool hack but I somehow feel like it should not work.

hiAndrewQuinn• 2 years ago

I'm somewhat of the same mind, but I'm fairly sure a study of Operating Systems: Three Easy Steps would get me over the hump. There's no actual reason to suspect this thing would like, sidestep normal process management or memory virtualization or something and run amok... I think.

alganet4• 2 years ago

It should work. We should have portable executables across many operating systems and architectures. The other expectation is scary.

cpach2• 2 years ago

There’s also Java, Python, Ruby, NodeJS etc etc.

3836293648• 2 years ago

Working across unices is reasonable. Adding Windows is less so

duped• 2 years ago

Why? Hardware and software architectures often have radically different needs from their loader, it doesn't make sense to have one format to rule them all. And any format flexible enough to support all use cases would just be a container for other formats, and in practice, loaders would ignore the variants that they don't/can't support so developers would need to care about porting things anyway.

ELF is pretty stable across many architectures and platforms but it's proven to be inflexible enough for plenty of applications where people develop their own, or alter it in varying ways.

mgaunard• 2 years ago

It doesn't really work, the libc is widely non-conforming.

I'd rather have a proper interface designed for portability rather than a hacked-together POSIX-but-not-really.

Conscat• 2 years ago

Programmers already deal with POSIX-but-not-really any time they deliver software for a Linux. At least this has the same quirks everywhere.

asveikau6• 2 years ago

Years ago I read some of the author's posts (they're active on HN too iirc) about it, and it seemed to me like they were relying heavily on the internals of the loader or dynamic linker on each platform, such that a new release for any given OS could conceivably break your binary.

It's probably a very fun project to hack on but I would advise against distributing the binaries and expecting them to work over the long term.

TimTheTinker4• 2 years ago

To my recollection, at least on *nix operating systems, they got some changes made to the POSIX standard to formalize behavior the binaries rely on. So going forward, mere POSIX compliance and ongoing ABI compatibility guarantee the binaries will continue to work on *nix operating systems.

On Windows, backwards ABI and executable compatibility has always been an extremely high priority, so I think the danger of future breakage is low.

Neither of those speak to macOS, but maybe someone who knows more can help clarify.

boricj2• 2 years ago

You can commit a lot of sins with ABIs if you throw the academic books into the fire.

For example, on my side I'm developing tooling that allows one to delink programs back into object files. This allows me to commit a whole bunch of heresy according to CS101, such as making a native port of a Linux program to Windows without having access to its source code or taking binary code from a PlayStation game and stuffing it into a Linux MIPS program.

When you're doing that kind of dark magic, it's one of those "know the rules so well you can break them" situation. Instead of following the theory that tells you what you should do, you follow the real-life world that constrains what you can do.

Aerbil313• 2 years ago

It's a cool hack but I somehow feel like it should not work.

It's rather obvious. It's bandaid that lets us continue on with the current outdated paradigms. Not a real innovation.

auggierose4• 2 years ago

Is there a UI library that goes with that?

DrNosferatu2• 2 years ago

A Web Interface for Cosmopolitan Apps would be a great bridge for an UI.

auggierose4• 2 years ago

I am looking for an Electron replacement for writing my apps. Could cosmopolitan be the basis of something like that, or is that not really its purpose?

lxgr• 2 years ago

Llamafile does that (i.e. it hosts a local web server that presents a GUI).

BiteCode_dev• 2 years ago

I would use redbean, from the same author, and use HTML.

auggierose4• 2 years ago

Impressive stuff. Unfortunately, Lua. From a user point of view (well, at least this user), TypeScript would be a better fit.

phkahler• 2 years ago

> I would use redbean, from the same author, and use HTML.

Wait... Could we build Solvespace web version and bundle it into that and have a completely portable CAD program that runs locally but in the browser?

pixelpoet2• 2 years ago

I checked two commits and they both had exactly the same number of chars per line, e.g. https://github.com/jart/cosmopolitan/commit/c4c812c15445f5c3...

I enjoy doing this too sometimes and don't find it too difficult, but damn...

whenlambo• 2 years ago

https://www.youtube.com/watch?v=Y65FRxE7uMc

lioeters• 2 years ago

I have found my people.

cgh• 2 years ago

Amazing, thank you for this.

pixelpoet2• 2 years ago

I thought I was alone!! wipes tears of joy

bdjsiqoocwk• 2 years ago

Well. Lorem Ipsum vs Donald Knuth. I think they both suck. I have a different hero. Linus Torvalds. He IS good AND actually delivers.

I enjoyed the video. However, both the author and Donald Keith strike as two people who are mainly motivated by aesthetics (which is supremely ironic as they almost define themselves as being the opposite of Loren Ipsum). I like people who deliver results.

I think the author mostly likes the appearance of someone who pays attention to detail. He says that in a movie if it's obvious that the coffee cups are empty, he notices and bothers him. 3 minutes later in his video he has a guy using his laptop at the beach where glare would make it impossible to see anything on the screen.

lproven2• 2 years ago

Back in the pre-GUI era, I did this with all my posts and comments online. All fully-justified, both margins, without any extra whitespace: just careful choice of words, and occasional (but always normal standard) abbreviations.

It was fun, once practised it barely slowed me down, and when people noticed it blew their minds.

Then along came GUIs and proportional fonts and made it all invisible. :'(

eesmith3• 2 years ago

Michael Hart, founder of Project Gutenberg, was known for this style.

https://en.wikipedia.org/wiki/Michael_S._Hart#Writing_style

See examples at http://hart.pglaf.org/ .

neerajk• 2 years ago

We'll give "real actual bug" a pass :)

edp• 2 years ago

There is a complete super metroid walkthrough using this technique : https://gamefaqs.gamespot.com/snes/588741-super-metroid/faqs...

ramses0• 2 years ago

cfg.linebreaks: LLM-plain-tex

tiffanyh2• 2 years ago

It's a bit easier to do when you don't limit yourself to exactly 80-char per line

  Those commits were 
  72-chars per line.
  Which is easier to
  do than EXACTLY 80
  character per line

aidenn02• 2 years ago

72 characters per line is a common standard for git commit messages. If you use vim, the syntax highlighting will nudge you to exactly that number.

https://tbaggery.com/2008/04/19/a-note-about-git-commit-mess...

layer8• 2 years ago

72 characters is the standard limit for Git though. This convention has a long tradition from the text-mode days of email and Usenet in 80-column terminals/screens, where 72 characters allowed for the addition of a reasonable number of quoting levels (indicated by prefixing quoted lines with “>”), and/or for other line markers, before exceeding the screen width and having to reformat.

Nevin1901• 2 years ago

The "actually portable executable header", fun to read through: https://raw.githubusercontent.com/jart/cosmopolitan/1.0/ape/...

dirkc• 2 years ago

I just scrolled for the images, I don't know about reading assembly for fun :)

j16sdiz• 2 years ago

in release note:

... , it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format ...

Did POSIX really _approve_ this? if yes, when?

xiaq• 2 years ago

Look for “POSIX” on https://justine.lol/cosmo3/

oguz-ismail3• 2 years ago

POSIX now says:

The input file can be of any type, but the initial portion of the file intended to be parsed according to the shell grammar [...] shall not contain the NUL character.

But the initial portion of, https://cosmo.zip/pub/cosmos/bin/cat for example, does contain the NUL character.

zbendefy• 2 years ago

wouldn't targeting webassembly provide something similiar? how would this compare to targeting WASM and running a runtime like wasmer?

optymizer• 2 years ago

WebAssembly is bytecode that runs on a virtual machine.

Cosmopolitan is a libc replacement. Your program is still compiled to real CPU instructions, which execute much faster.

tiffanyh2• 2 years ago

I assumed OpenBSD introducing pinning syscalls would prevent Cosmo from working.

I guess not.

https://news.ycombinator.com/item?id=38579913

geokon3• 2 years ago

It'd be fun/great if someone made Graal Native work with it. Then things could come full circle and I could compile my Clojure/Java binaries to run everywhere (but now without a VM)

simcop2387• 2 years ago

Forget which one, but there's a WASM runtime that's working on running on Cosmo-libc, combined with the WASM graal native port that's going... https://github.com/oracle/graal/issues/3391

nilslice• 2 years ago

We added Cosmo platform support to WAMR[0], to enable our Hermit[1] compiler and runtime.

[0] https://github.com/bytecodealliance/wasm-micro-runtime/tree/...

[1] https://github.com/dylibso/hermit

sgammon3• 2 years ago

work is ongoing

geokon3• 2 years ago

I'm quite fuzzy on the technical details - but do you think it'd ever lead to JavaFX running everywhere? It was a while before JFX worked with Graal Native

I'd like to someday be able to easily distribute my little GUI apps (for scientific applications) in a compile-once kind of way. Asking users to install a JVM runtime has turned out to be unworkable (everyone installs the Java 8 runtime b/c that's what comes up first on Google and then the apps silently crash/don't-work and users are confused)

ianopolous• 2 years ago

https://github.com/oracle/graal/issues/8350

metadat2• 2 years ago

Too bad it looks stuck since about a month ago (https://github.com/oracle/graal/issues/8350#issuecomment-210...), needs some attention from @jart.

superkuh• 2 years ago

Any idea if there will ever be a fix for the majority of linux distributions that cannot run cosmopolitan C applications without the bin fmt fix?

meisel• 2 years ago

Although this project is undoubtedly very cool, and maybe simplifies build processes by having a single binary, is there any other reason to use it? How does it compare in terms of performance, static linkability, standards conformance, etc. with musl and glibc? I’m curious because I’m picking a libc at the moment for my project.

doctorpangloss• 2 years ago

How does it compare in terms of performance

I tried rsync for Windows from its "cosmos" pile of binaries (https://github.com/jart/cosmopolitan/releases/download/3.3.1...), and it was too slow to be usable compared to WSL rsync.

For many other utilities I use Busybox for Windows (https://github.com/rmyorston/busybox-w32) which is well maintained, fast, and its maintainer is also very responsive to bug reports.

jart9• 2 years ago

You're basically just benchmarking the WIN32 filesystem versus a Linux VM in that case. The Windows file system is famously slow. It'd make more sense to compare a Cosmo binary running on WIN32 with a WIN32 native program. For example, Emacs has a WIN32 port where they wrote all the Windows polyfills on their own, and because they aren't experts on Windows, the official GNU Emacs Windows releases go a lot slower on Windows than if you just compile Emacs from source on Linux using Cosmo and scp the the binary over to run on Windows. See https://justine.lol/cosmo3/

tracker1• 2 years ago

I don't know enough to compare to musl, but I suspect being ISC licensed (similar to musl's MIT license) allows for static linking without upstream impact such as GPL.

Would be curious regarding features and performance comparisons with musl, which seems to come up a little short compared to say gnu libc.

saghm• 2 years ago

I suspect parent commenter isn't just asking about static linking from the perspective of licensing but about technical feasibility; at least part of the reason people use musl is that static linking glibc isn't well supported.

jart9• 2 years ago

Cosmo author here. We used to build our code with musl-cross-make. Once Cosmopolitan got good enough that we could compile GCC using Cosmo, our build latency dropped in half. https://x.com/JustineTunney/status/1726141024597324189

Cosmopolitan Libc is 2x faster than Musl Libc for many CLI programs like GNU Make and GCC because it has vectorized string libraries like strlen(). Musl won't merge support it on x86. I even mailed Rich patches for it years ago.

Cosmopolitan's malloc() function is very fast. If you link pthread_create() then it'll create a dlmalloc arena for each core dispatched by sched_getcpu(). If you don't use threads then it'll use a single dlmalloc arena without any locking or rdtscp overhead.

Cosmo has pretty good thread support in general, plus little known synchronization primitives from Mike Burrows. You can do things like build a number crunching program with OpenMP and it'll actually run on MacOS and Windows.

Cosmopolitan plays an important role in helping to enable the fastest AI software on CPUs. For example, Mozilla started an open source project called LLaMAfile a few months ago, which runs LLMs locally. It's based off the famous llama.cpp codebase. The main thing Mozilla did differently was they adopted Cosmopolitan Libc as its C library, which made it easy for us to obtain a 4x performance advantage. https://justine.lol/matmul/ I'm actually giving a talk about it this week in San Francisco.

yjftsjthsd-h5• 2 years ago

I would argue it's less about the build process and more about the user experience - no install, no "pick the right binary for your platform", just "download this file and run it". I don't think it's literally a static binary, but on anything but a `FROM scratch` container it might as well be.

Also I think there were some numbers showing that it sometimes had better performance than alternatives, but I can't seem to find the post right now.

nox1012• 2 years ago

The number of people this effects seems like it must be pretty small. There's the subset of people that have more than one OS. There's the subset of those that use command line tools (this explicitly doesn't do GUIs). There's the subset of those that would even think about some convenience of downloading a single binary command-line tool for multiple-OSes rather than use (a) something compiled specifically for that OS or (b) using something specific to that OS (rmdir on windows, rm on linxu/mac).

I use git, I've never felt especially put out that it's a different executable on mac, windows, and linux (situation "a"). I also just use what's common for the OS I'm on since there's so may other differences (C:\foo/bar) vs (/foo/bar) etc... (situation b)

1vuio0pswjnm72• 2 years ago

static musl make is at least 4x smaller than cosmo make

cosmo make probably works on at least 4x the number of operating systems

   # cd /usr/local/bin
   # file make
   make: /usr/local/bin/make: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped
   # stat -c %s make
   297104
   # cd
   # tnftp -4o make https://cosmo.zip/pub/cosmos/bin/make
   # stat -c %s make
   1397059

simjnd2• 2 years ago

The file size is larger, but it can use less memory than an x86-only Linux binary for musl libc: https://justine.lol/cosmo3/

stephc_int13• 2 years ago

Is there a plan to distribute the cosmopolitan libc without all the APE stuff?

For now it seems to be more like a framework+toolchain than a standalone library.

aidenn02• 2 years ago

I've been following it for almost 4 years now and I still do not know how serious of a project Cosmopolitain/αpε is.

TimTheTinker4• 2 years ago

The author is certainly very serious about writing highly distributable production software that works and will continue to work far into the future.

It's already being used for serious things - Mozilla's llamafile and redbean are two examples that come to mind.

tarruda• 2 years ago

I wonder if it is possible to have a 100% portable qemu build with this. Would be funny to have a usb stick with qemu + VM that runs on any OS.

jart9• 2 years ago

blink is almost as good as qemu-user and it works great compiled with cosmocc. https://github.com/jart/blink

modeless• 2 years ago

The commit message for the first commit there says "Actually Portable Executable now supports Android". I assume this means that the bare executables can run on Android kernels, not that there's any support for installing APEs as Android apps. But it seems possible for that to eventually work! Is that a goal of the project?

fusslo• 2 years ago

Never heard of Cosmopolitan before, but can I just say I really appreciate that the FIRST paragraph is a description of WHAT Cosmopolitan is and does.

There's way too many readme's that just jump in and assume the reader already is familiar with the project. it's a pet peeve of mine

kortex• 2 years ago

Looking forward toward somebody hooking together Python in APE [0], something like pex [1]/shiv[2]/pyinstaller[3], and the pants build system [4] to have a toolchain which spits out single-file python executables with baked-in venv and portable across mainstream OSes with native (or close enough) performance.

0 - https://news.ycombinator.com/item?id=40040342

1 - https://docs.pex-tool.org/

2 - https://shiv.readthedocs.io/en/latest/

3 - https://pyinstaller.org/en/stable/

4 - https://www.pantsbuild.org/

WaxProlix• 2 years ago

You can probably generate C code from Python now with Nuitka and pump that into this Cosmopolitan tool, today, to get that?

https://nuitka.net/

sangnoir• 2 years ago

I gave up trying to run cgo-compiled Go on an old appliance[1] the version of libc was simply too old. Has anyone ever successfully built cosmopolitan-flavored cgo binaries? I see Cosmopolitan libc supports Linux 2.6, so I'm hopeful.

1. Linux 2.x on little-endian MIPS.

throwaway20372• 2 years ago

From here: https://github.com/jart/cosmopolitan/releases/tag/3.5.0

Justine wrote:

    > We've made a lot of progress reinventing the C++ STL.

What is the motivation behind this? Try to reduce (compiled) binary size? Naively, I am surprised that the C++ STL from Clang isn't OK to use, including the license. Or is this a clean room impl thing?

EDIT

To be clear: I mean no disrespect with this question. This is an amazing project.

EDIT 2

Ok, I just spotted this commit message:

https://github.com/jart/cosmopolitan/commit/c4c812c15445f5c3...

    > We now have a C++ red-black tree implementation that implements standard

template library compatible APIs while compiling 10x faster than libcxx.

Can someone explain to an average C++ programmer like me how they make it compile 10x faster !?

simjnd2• 2 years ago

The rationale is explained in this README [1], but it basically boils down to transitive dependencies and the amount of checks the compiler has to do. From linked README:

"If we `#include <string>` for the LLVM libcxx header containing the `std::string` class, then the compiler needs to consider 4,800 `#include` lines. Most of them are edges it'll discount since they've already been included but the sum total of files that get included is 694!"

[1] https://github.com/jart/cosmopolitan/blob/master/ctl/README....

bdahz• 2 years ago

How does cosmopolitan support GPU-based applications?

nuc1e0n• 2 years ago

Can anyone point me to examples of this being used as a dependancy of other projects?

Cosmopolitan v3.5

Related Stories