jhgg

We just track power utilization.

Cool article :)

Yes, agreed, hence rarely a problem in practice ;)

Risk of deadlock is real if you have processes calling each-other in a cyclic way. e.g. process A sends GenServer call to process B, that then sends a GenServer call to process A to in order to handle the original call. However, process A is busy waiting on B to reply to it's initial call.

This is rarely a problem in practice however.

    > government passes law that requires companies to age verify users
    > said government provides no way to actually verify a human's age
    > hilarity ensues

Really cool! You looking to write Elixir as your main job?

This is not a model issue - you just have not properly implemented acoustic echo cancellation on your end.

The truth is this won't actually stop AI crawlers and they'll just move to a large residential proxy pool to work around it. Not sure what the solution is honestly.

Sweet! That's much more reasonable!

$5 to resize 1,000 images is ridiculously expensive.

At my last job we resized a very large amount of images every day, and did so for significantly cheaper (a fraction of a cent for a thousand images).

Am I missing something here?

IMO, it's not. All the UI boils down to: https://github.com/zhangchenchen/query_gpt/blob/main/src/api...

Metal looks super cool, however at my last job when we tried using instance local SSD's on GCP, there were serious reliability issues (e.g. blocks on the device losing data). Has this situation changed? What machine types are you using?

Our workaround was this: https://discord.com/blog/how-discord-supercharges-network-di...

Hoping this leads to datadog launching a logging solution that does not cost an arm and a leg at scale.

Except this article is about how their efforts to prompt the LLM didn't end up working and how they used embeddings / vector search to filter out comments that the LLM generated based on user feedback.

Does that matter when you can just swap it for gpt-5-whatever at some point in the future?

Are you implying this technology will remain static in its capabilities going forward despite it having seen significant improvement over the last few years?

I did this measurement, and using time -v, the maximum resident size in KB comes out to 440,424 kb for 1m tasks, 46,820 kb for 100k, and 7,156 kb for 10k.

When I worked at Discord, we used BEAM hot code loading pretty extensively, built a bunch of tooling around it to apply and track hot-patches to nodes (which in turn could update the code on >100M processes in the system.) It allowed us to deploy hot-fixes in minutes (full tilt deploy could complete in a matter of seconds) to our stateful real-time system, rather than the usual ~hour long deploy cycle. We generally only used it for "emergency" updates though.

The tooling would let us patch multiple modules at a time, which basically wrapped `:rpc.call/4` and `Code.eval_string/1` to propagate the update across the cluster, which is to say, the hot-patch was entirely deployed over erlang's built-in distribution.

Having a potentially untrusted client connect to the erlang node as a c_node (which seems to be what zerl does) is not a good idea generally, as connecting that way essentially allows the client to execute arbitrary code on the server.

https://github.com/discord/sorted_set_nif

A content-defined chunking scheme allows for you to build a pretty efficient game patcher where you can update between any version of the game (forwards and backwards) and doesn't require a bunch of server-side shenanigans.

League of Legends blogged about their updater here: https://technology.riotgames.com/news/supercharging-data-del...

I built something strikingly similar at my last job around the same time as well. Our server-side code was remarkably simple, just holding manifests (list of files and their chunk hashes) + a GCS bucket where all the chunks were stored (filename was just the hash of the chunk, making it content addressable.) Patching just involved the client understanding which chunks it already had, and which chunks needed to be downloaded from the server, or moved between files. Uploading a "new version" of the game to the server was also simple, just figure out what chunks were not in seen in the prior N versions of the game by looking at their manifests, and uploading them to the server, along with a new manifest describing the final contents of the game.

Given updating to a new version was just a matter of downloading new chunks, or re-arranging chunks that already existed on disk, going from one manifest to another did not depend on the new version being the successor of the prior version.

The majority of a game patch isn't actually the executable (where content-defined chunking wouldn't work well) instead were game assets like textures, audio files, etc..., which is why this strategy worked really well.

Yeah it just worked a lot better on scylla.

No cache. Just read coalescing. There is a big difference. Coalescing just ensures that while a query is executing if an identical query arrives, rather than sending the same query as an already executing query to the database it will wait for the existing query to complete and duplicate the result. If after this the same query arrives again, it will be issued against the database.

This means we don't have to deal with cache invalidation/consistency issues while also being able to handle thundering herds, for example a large server pinging @everyone and having a bunch of people click into the channel or launch their apps in response.

We did not change schema or partitioning strategy.

Scylla just eats all the ram it can with cache. So it's hard to say really. On Cassandra we allocated half the ram to the JVM which it gladly used up and left the other half to the OS for disk cache. On Scylla, since it uses direct io, there is no need for OS disk cache.

Read traffic is much higher than write traffic due to mobile clients needing to sync chat history more often as their sessions are much shorter lived. Also search queries execute 1 query per result. And don't forget people doing GDPR data dump requests. It adds up.

Nope. Didn't change the schema, mainly added read coalescing and used ICS. I think the big thing is when Scylla is processing a bunch of tombstones it's able to do so in a way that doesn't choke the whole server. Latest Scylla version also can send back partial/empty pages to the client to limit the amount of work per query that is run.

There are ballpark of a few hundred million discord servers... do you really want to run that many Postgres instances? And even so what do you do about DM/GDMs? Easier to just run one big mega cluster for messages.

Fwiw the benchmarked numbers are for writing very small rows. When doing the messages migration, with no read traffic, and the cluster/compaction settings tuned for writes we only managed approx 3m inserts/sec while fully saturating the Scylla cluster.

Yes, we wanted to migrate all our data stores away from Cassandra due to stability and performance issues. Moving to something that didn't have those issues (or at least had a different set of less severe issues) while also not having to rewrite a bunch of code was a positive.