aidanhs

Fair enough, would love to see another writeup on the performance you observe even if it fails - in practice numbers are hard to come by.

I've recently been going down the rabbit hole of creating a "fast start dev env" and it's interesting to see how this article differs from other approaches (codesandbox has some fantastic blogs, the fly.io blog on sprites has interesting pointers, e2b and daytona are related open source tools). Everyone has a different solution based on their tradeoffs.

I thought the memory snapshotting part in particular was clever since most container based systems don't bother (VM/firecracker based ones can use UFFD and call it a day), but by having emulated syscalls you can actually do single-process restore pretty well.

I am a bit dubious of the use of fuse (though it clearly works well!), and I wonder if ublk (what I ended up using) might alleviate some of the pain/magic in fuse tuning. I'd personally also be looking at forking gvisor to take a memfd which you enable UFFD on for the page loading (I have some firecracker patches where I do the same). It's nice because you can optimistically push pages, rather than waiting for the requests to come in. The series of three codesandbox blog posts are good background reading.

I'm super interested to hear more on anything you can share about your projects, or the niche of gov projects you're aware of - I've been doing some work with gov and haven't seen this requirement yet, so want to be prepared if it does come up.

(contact details in profile if you prefer)

One annoying part of using chroot if you're creating them on the fly is teardown - you have to manually invoke umount, and also take care to get this right for partially created chroots (maybe you detected an error after mounting proc, in the process of getting other files in place).

This was my original motivation in creating machroot (mentioned elsewhere in this thread) and having it use namespaces.

As a number of comments have noted, there are a bunch of different axes that chroot could be 'better' on - e.g. security and sandboxing.

I wrote https://github.com/aidanhs/machroot (initially forked from bubble wrap) a while ago to lean into the pure "pretend I see another filesystem" aspect of chroot with additional conveniences (so no security focus). For example, it allows setting up overlay filesystems, allows mounting squashfs filesystems with an overlay on top...and because it uses a mount namespace, means you don't need to tear down the mount points - just exit the command and you're done.

The codebase is pretty small so I just tweaked it with whatever features I needed at the time, rather than try and make it a fully fledged tool.

(honestly you can probably replicate most of it with a shell script that invokes unshare and appropriate mount commands)

Out of curiosity, what would you say the current state of the art is for full compilable decompilation? This is something I have a vague interest in but I'm not involved enough in the space to be on top of the latest and greatest tooling.

Interesting, that's helpful, thanks - so with the eventual arrival of memory64 and assuming I only wanted to target desktop systems and assuming browser implementations permit large allocations (e.g. 8GB) - large 64bit apps could work fine. I have a use case for this I've been poking at for a bit, but implementing my own version of cheerpx would be a lot of work, maybe I'll just wait!

On open source - I can only give you feedback as an outside fresh pair of eyes :) I incorrectly interpreted that it was full stack OSS based on the overall blog post 'vibe' and had to deliberately double check because I was aware of cheerpx beforehand. Perhaps it's just me. I look forward to the cheerpx blog post!

All the emulation of desktop machines in WASM I've seen so far have been for x86 - do you think there are significant additional hurdles for x86_64? Or is it just a matter of time?

Separately, one bit of feedback - it's cool that webvm is open source, but I think it's fair to ask you to be upfront that cheerpx itself is not (which is fine!) in the blog post itself where you talk about webvm licensing. If I wasn't already familiar with the wasm emulation space I would have felt rather misled.

Right, because Anyscale found a niche that distributed compute matters in (AI) and built great libraries/hosted platforms/services around that. I would venture that the money they make from people who pare back things to just ray core is ~0, which is why it's open source.

Put another way - building such a platform doesn't preclude commercial success, but (at least for us) it isn't sufficient. Fly.io might be able to pull it off if they want to explore that direction imo.

Fwiw if you dig around in the ray core codebase (as I did when I was doing competitor analysis years ago) you can use the core C code from other languages to build such a platform for Rust if you like - they had Java and C++ interfaces at the time, but I haven't looked in the last 5 years.

The company I work at (Hadean) used to have this as a product - think erlang-like multi machine IPC, with automatic acquisition of cloud resources and language integration for Rust, C, C++, Python. Pretty easy to point it at some machines and get them running a distributed application (as in simulation or big data).

But infrastructure for developers is hard to make money with - developers like to build it themselves and people holding the purse strings point at kubernetes and say "that's free". So we just use it as an internal platform for a distributed simulation engine and it works pretty well.

I did an analysis of removing it (it's a lot of bespoke code that we have to maintain for something that isn't our actual product) and I think you could probably implement something on top of Nomad that's close enough...but then Nomad went BSL and Kubernetes is a big complexity shift.

So...if anyone knows of something out there let me know, I'd love to be able to use it outside of work :)

First a minor quibble: if you're talking about 'Nix data', then starting the conversation by talking about 'Nix-the-language' is rather misleading.

That aside, Nix allows you to create infinite datastructures, e.g.

    $ nix eval --expr 'rec { z = { a = z; i = 5; }; }.z.a.a.a.i'
    5

But even if JSON did have some way of handling datastructure 'loops', it's still not helpful because of laziness. You almost never want to eagerly evaluate a Nix expression to produce what you seem to term 'Nix data', because you'll invoke the `derivation` built in function to create paths you never actually reference - this is why laziness is such an important property of Nix.

So I'm still not clear what user-facing part of Nix is isomorphic to JSON. If it's just "Nix types [0] are similar to JSON types" then...sure.

[0] https://nixos.org/manual/nix/stable/language/values

Can you elaborate on what you're getting at?

Syntax-wise it's about as similar to JSON as Erlang expressions are (i.e. superficially similar in some cases).

Semantics-wise I've personally found any superficial similarity to JSON to be actively unhelpful in understanding because of some important processing differences (e.g. paths, laziness).

Of my series of PRs, I suspect the third (i.e. https://github.com/zellij-org/zellij/pull/3043) is most likely to have an effect. But if it does it'd only be as a side effect unfortunately - my focus was on fixing lag with splitting of extremely long lines.

From what I saw while making my changes, that area of the code has a bunch more possible optimisations, but it's 'good enough' for me at this point so I'm not planning to continue pulling at the thread right now. If you wanted to look yourself, I left the script I used for benchmarking and profiling in https://github.com/zellij-org/zellij/issues/2622#issuecommen...

I didn't believe you that it was broken, but you're right - very disappointing. For anyone interested, the bug for it being is at [1] (reported mid 2021).

The build failure is easy to fix, so I created a repo at [2] which builds a program against a glibc with static nss. I verified with strace that it does indeed check nsswitch.conf and try and load dynamic libraries (I'd at least submit my patch [3] for the build failure but I find mailing lists to be a hassle)

All this said, I wouldn't call it undocumented - it's documented in the `configure --help` itself as well as the online version [4], and it has an FAQ entry [5].

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=27959

[2] https://github.com/aidanhs/gcc-static-linking

[3] https://github.com/aidanhs/gcc-static-linking/blob/1f04425e2...

[4] https://www.gnu.org/software/libc/manual/html_node/Configuri...

[5] https://sourceware.org/glibc/wiki/FAQ#Even_statically_linked...

Hadean | Rust Engineer | London (flexible/hybrid) or REMOTE (UK) | Full-Time

Hadean are backed by the likes of Epic Games. Our speciality is in spatial compute - we’ve built a massive-scale distributed simulation engine and a connectivity layer to plug thousands of users into a single world. We’re using these to provide the infrastructure and computational power to build, run and monetise the Metaverse.

Our core (internal) platform, connectivity layer and some higher level components are written in Rust and we’re looking for people to work on the design, implementation, and maintenance of our products.

Check out our careers page at https://hadean.com/jobs/

(for context - I'm not interested in first class node support)

This seems pretty cool. I particularly like how 'gradual' it seems to be relative to things like Bazel, i.e. you can take some shell scripts and migrate things over. I did have a play and hit an initial problem around project caching I think, which I raised at [0].

One comment, from the paranoid point of view of someone who has built distributed caching build systems before is that your caching is very pessimistic! I understand why you hash outputs by default (as well as inputs), but I think that will massively reduce hit rate a lot of the time when it may not be necessary? I raised [1].

Edit: for any future readers, I spotted an additional issue around the cache not being pessimistic enough [3]

As an aside, I do wish build systems moved beyond the 'file-based' approach to inputs/outputs to something more abstract/extensible. For example, when creating docker images I'd prefer to define an extension that informs the build system of the docker image hash, rather than create marker files on disk (the same is true of initiating rebuilds on environment variable change, which I see moon has some limited support for). It just feels like language agnostic build systems saw the file-based nature of Make and said 'good enough for us' (honorable mention to Shake, which is an exception [2]).

[0] https://github.com/moonrepo/moon/issues/637

[1] https://github.com/moonrepo/moon/issues/638

[2] https://shakebuild.com/why#expresses-many-types-of-build-rul...

[3] https://github.com/moonrepo/moon/issues/640

Readers please note that sccache isn't just for Rust, and isn't just caching!

It has a number of features, combining capabilities of ccache/distcc/icecream for C, C++ and Rust...along with some unique things that I've not seen in other tools. My comment at https://news.ycombinator.com/item?id=25604249 has a summary.

I recently refactored some traits that had a lot of repetitive noise in the generic type parameters (which end up being reflected in function signatures etc). I went about this without really being overly aware of GATs and their status, but the code that I naturally ended up with required enabling them.

So from my point of view, I didn't really need to understand what GATs were bringing - it's a lifting of restrictions that would have previously felt arbitrary.

In the context of the GDPR, I just want to remind people of this thread where a HN user invokes their rights in order to make Spotify back down on a change that would have locked user playlists into their service for no good reason - https://news.ycombinator.com/item?id=24764371

(can't be 100% sure this is what made Spotify change direction, but it seems likely)

This is exactly what I was hoping for! If you just copied and pasted this into the documentation directly, that'd be more than enough. Thanks for writing it out so clearly and creating the issue.

First off, I love zstd and thanks for your work - I've used it multiple times with great success.

My question is...what's up with zstd compression levels? It seems to be impossible to find documentation on what the supported compression levels are, and then there are magic numbers that make things more confusing (I think last time I checked, 0 means 'default' which translates to level 3?)

My notes when I was trying to track through the minimum compression level through multiple header files seemed to indicate it's MINCLEVEL...which appears to be -131072? But it starts talking about block sizes and target length, and I'm not entirely sure why they would relate to a compression level.

  # define MINCLEVEL  ZSTD_minCLevel() [0]
  int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; } [1]
  #define ZSTD_TARGETLENGTH_MAX    ZSTD_BLOCKSIZE_MAX [2]
  #define ZSTD_BLOCKSIZE_MAX     (1<<ZSTD_BLOCKSIZELOG_MAX) [3]
  #define ZSTD_BLOCKSIZELOG_MAX  17 [3]

Basically, is there a decent reference I can be looking at here?

(disclosure: I don't use tasker and am not associated with them, I do work on something in the similar space of "compute without servers")

First, you're coming at this from the wrong perspective. The OP is about alternatives to Heroku - the starting position is "I don't want to use servers". There are a number of rational reasons for this, including OS maintainence and the mental overhead of setting up continuous deployment from your repo (ansible to configure cron? a self-hosted PaaS?). "Don't worry about servers" doesn't just make sense in principle, it's also quite popular in practice - just consider the uptake of things like serverless.

So, starting from "no servers", how do you run something on a regular basis? Let's say "send me a message on telegram daily with some info". Serverless and similar offerings are useless for this, they're a different model (request/response). I did research on "hosted cron" at the time and the options were pretty terrible - there's definitely space here for tasker.

FWIW, this is coming from someone with a default position of "self-hosted only". For my own use, I tried out cron and airflow - they're both annoying and I settled on bgproc [0] as least-worst on my already-existing personal server. I really wanted to move to something hosted by the time I was done (tasker didn't exist).

[0] https://github.com/seanbreckenridge/bgproc

In the concrete context of the post you're replying to, you seem to be saying that women are 'vulnerable people'? I'd suggest picking a definition of 'vulnerable people' that doesn't mean pushing away half the world's population.

(I'm also a little bemused at your implication of what an 'uncomfortable idea' is, as well as the implication that hackernews is the right place for this, as well as the assertion that you can fragment social networks to satisfy people while ignoring the effects of critical mass - but there's too much to unpack)

Your second paragraph is irrelevant in this thread (which discusses HN specifically).

Ad funding would never stop because people wanting to make money off content would never stop - it just becomes more subtle. Think sponsored posts, product placement, added bias.

Imagine a plain text blog run by an individual, good quality technical content, and a company comes and offers $10k to write a blog post about their product after trying it. If the author decides not to disclose the financial motivation, you now have a subtle biased funded advert that no adblocker will save you from.

Honestly, I struggled with this overall.

For context: I'm mostly unaware of actions of US presidents (beyond the broad sweeps based on left vs right) and I've at most listened to Trump and Biden maybe a couple of times for a total of ~1min in the last few years - just doing this exercise has probably at least doubled my exposure.

I found the text and voice mostly impossible unless the content clued me in. Video was a little easier because I know to look for teeth, video with sound was fairly easy.

Is there a generic trick for recognising faked voices without really knowing the original (similar to looking for teeth on videos)?

This is missing a lot of detail around libc (can't comment on the others), but was published ~1.5 years after musl was first released - I can imagine the subtleties being missed in a glibc monoculture.

To expand a bit:

1. dlopen has interactions with static linking that you should understand if you use both at the same time. Different libcs expose different symbols for libc functions, so building your dynamic library against one libc may make it incompatible with another. And two libcs in the same program (one from the dylib, one from the binary) is a recipe for a very bad time - imagine freeing a pointer allocated with a different malloc implementation, or a different layout of libc structs

2. the complexity with glibc is it invokes dlopen as part of NSS, a feature that gets used as part of looking up users, among other things (it does this to allow integration with LDAP etc). You can actually see warnings at link time if you statically link glibc but pull in symbols that use NSS. You can disable NSS if you like at build time of glibc itself [0] (i.e. not your binary)

3. musl doesn't have the complexities of NSS and can be statically linked happily by default (I suspect that's probably what musl is most used for)

[0] https://sourceware.org/glibc/wiki/FAQ#Even_statically_linked...

I've created ~five new projects over the past year using SQLite and I've got into the habit of creating tables like this:

  CREATE TABLE tIssue (
    id   INTEGER PRIMARY KEY NOT NULL CHECK (typeof(id) = 'integer'),
    col1 BLOB NOT NULL                CHECK (typeof(col1) = 'blob'),
    col2 TEXT                         CHECK (typeof(col2) = 'text' OR col2 IS NULL)
  );

I'm very much looking forward to using strict tables.

I've never used or looked at Typesense (I've been perfectly happy as a Meilisearch user), but your characterisation of interacting with Meilisearch is so alien it makes me wonder if we've been looking at the same project.

Across the assortment of Meilisearch repositories, I've raised two PRs (one accepted, one rejected), five issues, one feature request and pinged one issue for an update.

Every single time the Meilisearch team has been responsive, communicative and generally a delight to interact with - there are very few projects I would consider better.

Just thought I'd throw in my experience.

Context: O3DE is an open source (Apache 2) AAA game engine based on an updated Amazon Lumberyard

I've had the opportunity to work with O3DE over the past few months at work and I'm very excited to see where it goes.

There are two pieces of context if you don't work in the games industry (when I came in from an 'open source' background these surprised me):

1. the best game engine tech in the industry is proprietary (the biggest players are Unreal and Unity, lots of studios have purpose-built ones), there's relatively little open source [0]

2. in a games studio, the workflow of your company orients around the workflows of the game engine [1]

These two things combine in quite unfortunate ways, to the point where EA mandated the use of its own in-house engine...with mixed results so far [2][3]. Smaller players have three options: a) suck it up, b) use an open source alternative (i.e. Godot), c) get caught in the tarpit of "build your own engine"

O3DE has an opportunity to shake things up - particularly because of how modular it aims to be. At Hadean, our interest is in the networking layer (to drop in as netcode for Aether, our distributed simulation engine) as well as the 'core' simulation layer (to integrate with Aether in order to scale O3DE games/simulations across multiple machines). While doing all this, we will probably 'get involved' and contribute, so that our usage doesn't drift too far from mainline.

Anyway, this is all very early days - it's just a developer preview, and the flows are a bit rough. It'll be a while before we see if this changes things (I'm hopeful!), but major credit to the people at Amazon who have made it happen so far!

p.s. a checkout with deps, plus fresh compile of O3DE, plus a new project that you've opened once (to process assets) will set you back 100-150GB at my last check - not unexpected if you're used to compiling Unreal!

[0] https://twitter.com/ID_AA_Carmack/status/1406427706980454406

[1] https://isetta.io/interviews/AmandineCoget-interview/#the-mo...

[2] https://twitter.com/kingcurrythundr/status/10837887204599726...

[3] https://screenrant.com/anthem-frostbite-engine-development/

I first came across this a while ago and after digging I found the way it worked (and possibly still works) behind the scenes is pretty cool: https://groups.google.com/g/emscripten-discuss/c/4Qw8OOgTvu0...

A while ago I was pondering getting emscripten working under Wine, but realised that it's not quite as simple as a typical program (because Wine relies on being able to just provide a compatibility layer for syscalls, so the compatibility layer for x86 would have to be provided by the port to emscripten).

Seeing this post, I wondered if you'd found some cunning way around this so downloaded the project. No, you've just gone straight ahead and implemented an x86 emulator! Ok, but how is wine able to run on bare metal? Oh right, you've actually implemented parts of linux, including most of the syscall interface, your own executable loader, parts of procfs and assorted special devices! Alright, but (as Alon asks) how does this actually render things? Ah I see, you've implemented your own display driver in Wine which, when methods are invoked, dispatches interrupts to the emulated CPU which then forwards and translates the interrupt to an equivalent sdl call (with opengl working a similar way, but with a dll)!

Wow!