Dave_Rosenthal

Great Q--it's weird that Apple, Snowflake, and the other big contributors have put essentially zero calories into documentation/website/etc.

There has been some movement lately, though. I'm not sure who exactly to thank, but "the community" created a new unofficial resource for getting going with FoundationDB: https://foundationdb.vercel.app/7.3/

They say, "our goal here is to improve PostHog as a product for our customers, not to expose or sell models trained on your data" but then don't actually list that as a limitation in the bulleted points.

AFAICT this now gives them default permission to train an LLM on your code (as Posthog telemetry data is inextricably tied to your code) use it, and even sell it if they wanted to (as it's not your data anymore, it's their model). Yikes.

Simple answer: There is no physical basis, it's style

Pedantic answer: Unless the light source has different colors on different sides

Complex answer: Kind of. Even a linear color fade (from reality) can turn non-linear (and therefore induce color effects) when pushed through a color grading pipeline. So if you count e.g. film emulation as a "physical effect", then yes.

The car guys have done it: https://www.youtube.com/watch?v=stfjVt0AbFU

As a boss-man myself, I’ve seen this “don’t let them take advantage of you” sentiment expressed in many discussions about comp and promotions, but I can’t really say I understand it. Am I just out of touch?

As I read it, the article is simply trying to help people understand what kind of work is valuable to a company and therefore what they should focus on to make themselves valuable. I presume that making yourself valuable pays dividends, including promotions! Somehow the idea of going to work and not trying your best because “you’re not getting paid … for that” just feels so cynical and divorced from how I’ve seen successful people grow and make big bucks in tech.

(And this is all a bit separate, of course, than the debate about whether staying at a company or job hopping is better for career trajectory.)

RTM was my TA at MIT for a CS/systems engineering course. It took the students until we did an assignment about the worm to realize who he was IIRC. The students thought it was very cool, but even then, as a TA covering the assignment, he didn't really talk about it.

Neat visualization. There is a street performer in my town that has them all memorized (and many international zip codes as well) and makes a fun show of it, asking people what zip code they are from and putting them on a map. Sometimes it takes him a second to recall the exact town name but he always knows instantly what city they are close to, helped no doubt by the locality of the mapping. Example video:

https://www.youtube.com/watch?v=eSFt38IS0QU

Sure, let's take a rug as an example. I don't think there is one breakpoint. I think there are a set of axis of quality you invest into, roughly sequentially, as you go up the price scale of objects:

- $50 - Something rug-shaped exists

- $100 - Durability

- $200 - Materials

- $500 - Comfort and design

- $1000 - Basic craftsmanship

- $2000 - Refinement of craft

- $5000 - Artistry & identity

- $10000 - Tradition

- $20000 - Mastery

- $50000 - Rarity/historical importance

- $100000+ - ?

Because most people don't cross-shop $20k rugs and $200 rugs, most people are focused on one or two aspects around their personal budget. The essayist mentioned being amazing by the craftsmanship and artistry (see scale above). A broke college student might just want something that holds up in their dorm room and see what materials it's made out of and comfort as meaningless and abstract. And a billionaire shopping for a rug for their office might take everything other than rarity/historical importance as a given and just be thinking about that.

I think there is a large cognitive bias to consider everything you can easily afford "tangible and important improvements" and everything you can't as "abstract"!

We thought of that demo with like a week or two to go before the event when we realized we needed something more visceral than a blinking cursor :)

So, it stayed with the world because it went semi viral in some tiny circles. But it didn’t stay with me because I personally learned little from it.

The thing that stays with me is all the things I learned spending years trying to implement an API you can describe in ~30 seconds really well. It’s a rare opportunity to be able to go so deep on a problem in a career.

The documentation is woefully out of date, sadly. Despite the code being in active development no one is touching the public docs. Though I don’t know for sure, that limitation was probably written something like 10 years ago.

Ha, well I met Scherer ~30 years ago in a high school math class and we’ve done three companies together, so you could say we’ve known each other for a bit :)

Well, I stand very much corrected since that's me in the video! I forgot that an early version ever made it out into the world.

It looks like it might be knobs (which can be changed via config file) called MAX_WRITE_TRANSACTION_LIFE_VERSIONS and MAX_READ_TRANSACTION_LIFE_VERSIONS now (defined in ServerKnobs.cpp)? It's in microseconds and probably needs to be synced client and server.

I don't know the details know, but it was definitely configurable when I wrote it :) I remember arguing for setting it to a default of 30/60 seconds we decided against as that would have impacted throughput at our default RAM budget. I thought might have been a good tradeoff to get people going, thinking they could tune it down (or up the RAM) if they needed to scale up perf later.

You might be able to, but are definitely not supposed to. The client is conceptually "part of the cluster".

There have been some 3rd party toy projects in the past years, but pretty sure nothing was ever released by FoundationDB/Apple relating to SQL (until this additional SQL interface on the Apple "record layer").

Thanks for the insight. A $XX million exa-data system is no doubt impressive :)

Another big problem I found with modeling real apps was the five second transaction timeout. This is not, as you might expect, a configurable value. It's hard-coded into the servers and clients. This turns into a hugely awkward limitation and routinely wrecks your application logic and forces you to implement very tricky concurrency algorithms inside your app, just to do basic tasks. For example, computing most reports over a large dataset does not work with FoundationDB because you can't get a consistent snapshot for more than five seconds!

I'm pretty sure that the 5-second transaction timeout is configurable with a knob. You just need enough RAM to hold the key-range information for the transaction timeout period. Basically: throughput * transaction_time_limit <= RAM, since FDB enforces that isolation reconciliation runs in memory.

But, the other reason that 5 seconds is the default is that e.g. 1 hour read/write transactions don't really make sense in the optimistic concurrency world. This is the downside of optimistic concurrency. The upside is that your system never gets blocked by bad-behaved long-running transactions, which is a serious issue in real production systems.

Finally, I think that the current "Redwood" storage engine does allow long-lived read transactions, even though the original engine backing FDB didn't.

FoundationDB's original promise was to combine a distributed storage engine with stateless layers on top to expose a variety of useful data structures (including SQL). The company was acquired before it could release the layers part of that equation. Apple open-sourced the core storage engine a few years later so FDB has kind of had a second life since then.

In that second life, the vast majority of the powerful databases around the industry built on FoundationDB have been built by companies making their own custom layers that are not public. This release is cool because it's a rare case that a company that has built a non-trivial layer on top of FDB is letting that source code be seen.

The group to which the FoundationDB storage engine itself appeals is fairly narrow--you have to want to go deep enough to build your own database/datastore, but not so deep to want to start from scratch. But, for this group, there is still nothing like FoundationDB in the industry--a distributed ACID KV store of extreme performance and robustness. So, yes, it's still the hotness in that sense. (As others have mentioned, see e.g. Deepseek's recent reveal of their 3FS distributed filesystem which relies on FDB.)

I was curious to see how audible square wave duty cycle would be (I figured not much) but sadly the audio examples are clearly broken on this page--choosing the same one multiple times gives totally different sounds.

Huh, I was actually on this page a few years ago, but iOS and MacOS quality has been super solid for me this past year. Anyone else feel this way? Judging by the nodding comments maybe I’m just the outlier?

You were talking to a slightly earlier company--this was post Viaweb.

I worked with pg in a three-person startup in a basement in Harvard square just before he started YC. I was an undergrad at MIT and met him after arguing with rtm (who was a TA for my class) about TCP backoff. It was called Aspra and we were trying to build an app development platform for mobile phones. The idea was that you could build an application that could "render" to either an IVR system, for dumb phones, or the kind of web page that a Motorola "smartphone" of the day could view. Ahead of its time, like Paul. It didn't go far, and not long after he started YC. Good thing he did, for the industry and many of the folks here! Anyway, the "20 years" got my attention because I didn't feel that old, and thought someone might like the little story.

Interesting article. I would have loved to see the quality ratings weighted by how many people drive the roads, not by road-mile.

I bet it would look a lot worse. It seems like low traffic roads out in the middle of nowhere are pretty decent, while the multi lane juggernauts in downtown that everyone spend their time on are disasters.

What no-one is pointing out is that LLMs have made almost as much progress on the first part of the request as the second! ChatGPT writes me a is_point_in_national_park function and points me to the relevant shapefile in like ~30 seconds. That's a few hundred times speedup of the "few hours" referenced in the comic.

That class scheduling one is a good place to grok the basics. Just extending the kind of approach introduced there will get you decently far.

The best resource after the tutorials is: https://github.com/FoundationDB/awesome-foundationdb

Unfortunately, I think a lot of the advanced 'tricks of the trade' (alternatives to use cases for long-running transactions, when exactly to dispatch to a cloud object store, how to migrate whatever schemas you end up creating, etc.) that all the big serious users of FDB are doing are not as well covered.

As a FoundationDB partisan, it's been great to see more and more people starting to build stuff with it. Someone recently told me that they think of FDB as "assembly language for databases". I liked that. I think it faithfully captures both the good and bad parts of FoundationDB:

1) You are starting with a blank canvas

2) You will be dealing with low-level primitives

3) You have complete freedom in what you build

4) It's fast if you use it right

For each one of these items you could look at it and say "that's awesome" or "yikes." You know which group you are in!

I hope more people in the 'awesome' group realize that using FoundationDB can transform the development of a new distributed data store from a multi-year project into a few weeks of hacking. This is because FoundationDB solves the scalability, fault tolerance, and ACID parts for free--the least creative and most time-consuming aspects.

A few comments:

- My understanding is that a gamma chirp is the established filter to use for an auditory filter bank--any reason you choose an elliptical filter instead?

- I didn't look too closely, but it seems like you are analyzing the output of the filter bank as real numbers. I highly recommend you convolve with a complex representation of the filter and keep all of the math in the complex domain until you collapse to loudness.

- I'd not bucket to discrete 100hz time slices, instead just convolve the temporal masking function with the full time resolution of the filter bank output.

- You want to think about some volume normalization step that would give the final minimized Zimtohrli distance metric between A and B*x, where x is a free variable for volume. Otherwise, a perceptual codec that just tends to make things a bit quieter might get a bad score.

- For fletcher munson, I assume you are just using a curve at a high-ish volume? If so, good :)

- Not sure how you are spacing filter bank center frequencies relative to ERB size, but I'd recommend oversampling by a factor of 2-3. (That is, a few filters per ERB).

Apologies if any of these are off base--I just took a quick look.

A lot of people saying stuff like sports or pottery. But I have a hard time thinking of those as art forms meant to be experienced through touch. What comes to mind for me is something like the "Tactile Dome" at the Exploratorium in SF--an exhibit for kids where it's dark and you move through it feeling different things with different sensations. I think food is also a good candidate.

One thought is that vision and hearing are all 1->many senses. That is, many people can stand around and experience one fireworks show, or one musician, or one painting. We've even invented ways of amplifying the "many" with speakers, projectors, etc. Perhaps this makes these art forms more commercially viable than touch, where a single person experiencing it excludes anyone else from the same experience? Maybe we need to invent a "touch multiplier" before the genre will take off?

+1 You're not missing anything. And there are plenty of very simple ways to just use a few "special destinations" to encode any move including possible promotions in a constant 12 bits per move and be very fast to encode/decode. The proposal in the article is just silly.

But really the whole thing is silly. 100 million games is a lot of chess, but at 1KB per "inefficient" PGN you are talking a whopping 100GB--big deal. (At 12 bits and, say, 80 half-moves on average, you are talking ~12GB.) Plus the article says that he is "IO-constrained" but shrinking game size isn't going to help with random lookups--a PGN is already below the page size.

If you really did care about best compression you would simply assign all legal moves an index and use a heuristic (chess engine) to figure out which moves were likely (i.e. good). In chess, it's not uncommon that good players are usually picking between only a few reasonable candidate moves. I wouldn't be surprised if good compression of human games yielded something like 2-3 bits per move.

With the "production-grade" part of the title, I was hoping to see bit more about scalability, fault tolerance, updating continually-changing sources of data, A/Bing new versions of models, slow rollout, logging/analytics, work prioritization/QOS, etc. It seems like the lack of these kind of features is where a lot of the toy/demo stacks aren't really prepared for production. Any thoughts on those topics?

I think we need to withhold analysis based on the 34 PPD number. In the article you can see that the rendering projection is quite distorted, with the apparent effect of using more pixels per degree for the center part. They don't account for this when calculating the 34 number. Of course a "fisheye" lens would also be "distorted" and allocate an equal number of pixels to each angle, so it's hard to tell by just eyeballing things. I wouldn't be surprised if the actual PPD number in the center, where it matters, is higher.