uvdn7

This is something AI would never take away from us.

It reminds me of Clay Christensen’s book How to Measure Your Life. In one of his talks, he talked about how companies get killed because they optimized for the wrong/short-term metrics. What we are seeing with AI could be a supercharged flavor of Innovator’s Dilemma, where organizations optimize a pre-existing set of success metrics while missing the bigger picture because some previous assumptions no longer hold.

I really like the article. It’s not trying to sell fear (which does sell); it doesn’t paint the leaderships as clueless. Nobody knows what is going to happen in the future. The article might be wrong on a few things. But it doesn’t matter. It points out a few assumptions that people might be missing and that is great.

Is the non-trivial amount of time significantly less than you trying to ramp up yourself?

I am still hesitant using AI for solving problems for me. Either it hallucinates and misleads me. Or it does a great job and I worry that my ability of reasoning through complex problems with rigor will degenerate. When my ability of solving complex problems degenerated, patience diminished, attention span destroyed, I will become so reliant on a service that other entities own to perform in my daily life. Genuine question - are people comfortable with this?

You are also not locked down in C++. There are already a handful of coroutine and async runtime implementations out there.

I’m not totally sure if this is a GOOD idea to add to the c++ standard

What are the downsides? Naively, it seems like a good idea to both provide a coroutine spec (for power users) and a default task type & default executor.

I am not a native speaker and I joke about my typos and grammar mistakes being the evidence that none of my code or post is AI generated. Sorry about the typos. I just fixed all the ones I can find. Hope it's better now.

Neat. It certainly makes oncall and maintenance easier! It is likely more resource efficient by e.g. minimizing idle compute, maximizing cache hit rate, etc.

Love it! I wonder if the team knew this explicitly or intuitively when they deployed the strategy.

We created a rule in our central monitoring and alerting system to randomly kill a few instances every 15 minutes. Every killed instance would be replaced with a healthy, fresh one.

It doesn't look like they worked out the numbers ahead of the time.

I was thinking the same. There are many mutexes out there, some are better at certain workloads than the rest. DistributedMutex and SharedMutex come to mind (https://github.com/facebook/folly/blob/main/folly/synchroniz..., https://github.com/facebook/folly/blob/main/folly/SharedMute...) Just like hashmaps, it's rarely the case that a single hashmap is better under _all_ possible workloads.

Math, sure - doesn't the understanding of physics also go through changes? Do we really understand the reality of the world; or how do we know our current understanding of it won't change?

The same happens in work place as well, when multiple people have similar ideas and it is unclear or impossible to credit who came up with the idea first. It is often the case that people with shared context had similar ideas independently when facing the same set of problems.

I have the impression that it is controversial about who invented the hybrid logical clock first as well. Although most people cite the Kulkani paper from 2015 I think?

Do people know who else claims to invent HLC first?

The biggest difference between automatically keeping an MV up to date vs keeping indices up to date is that the write amplification of the latter is a function of the index count you have, while the former is a function of data and query. It’s easy to come up with cases when users update a single db row, and you end up having to update millions of rows in a MV (eg the every row in the MV has a name “Rob” and Bob changes his name).

I read the timely dataflow which underpins materialize.com. It seems like we don’t necessarily need the support of loops, which timely dataflow allows, for regular SQL, which is a DAG of operators. It appears that as long as the database supports snapshot reads, one can have a push-based query execution to enable incremental MV updates. The problem, I think, is still in the write-amp-as-a-function-of-data, which is unbounded. It is very cool regardless.

The technique can be used for cache invalidation as well, given the data cached needs to be described in SQL, which seems reasonable.

Apple claims that its Weather App has integrated DS’s features. However, for a few times it’s ridiculously wrong. It said it’s cloudy when it’s raining onto the very phone, which I don’t remember happened to DS. Forecast for the next hour feels like as accurate as forecast for next month. I live in New England. I am going to miss DS a lot.

I think it boils down to cost. Digital content is extremely cheap to replicate and distribute; hence the quality of the content is assumed to be lower (and it’s usually the case). One doesn’t have to put in too much effort in producing digital content.

Now with ChatGPT, it’s even more so than ever. It’s practically free to produce digital content.

On the other hand, even by just printing out a doc increases the cost of replication and distribution significantly (compared to pure digital replication); hence the quality of the content is assumed to be higher.

Rust borrows c++’s modern memory model verbatim.

C++ is a language that keeps innovating besides many legit reasons to hate the language. Having more competition is good. I would expect more rust/{your favorite language} concepts to show up in future versions of c++ (once proven to be a good idea in most cases).

By this reasoning, shouldn't Tesla pay users instead to enable FSD and collect data for them?

And no abstraction is perfect, as by definition an abstraction hides some details from the layer beneath. A good abstraction is one that allows you to not look under the hood most of the time. One can be happily writing code in their favorite programming language until you want better performance and started looking into cpu caches, branch prediction, etc. which are "breaks" the nice abstractions provided by the OS and high level programming languages.

A bit off topic, Eli Bendersky's entire blog is absolutely amazing. I started writing a technical blog inspired by his. Most of his recent blogs seem to be about Go, which I don't know much about. But I found myself going back to some of his older blog posts many times.

It's not meant to be a perfect analogy. The replication analogy is mostly talking about the tradeoff between performance and cost. So it's less about "replicating" the ip addresses (which is not happening). On that front, maybe distribution would be a better term. Instead of storing a single piece of data on a single host (unicast), they are distributing it to a set of hosts.

Overall, it seems like they are treating ip addresses as data essentially, which becomes most obvious when they talk about soft-unicast.

Anyway, I just found it interesting to look at this through this lens.

This is a wonderful article. Thanks for sharing. As always, Cloudflare blog posts do not disappoint.

It’s very interesting that they are essentially treating IP addresses as “data”. Once looking at the problem from a distributed system lens, the solution here can be mapped to distributed systems almost perfectly.

- Replicating a piece of data on every host in the fleet is expensive, but fast and reliable. The compromise is usually to keep one replica in a region; same as how they share a single /32 IP address in a region.

- “sending datagram to IP X” is no different than “fetching data X from a distributed system”. This is essentially the underlying philosophy of the soft-unicast. Just like data lives in a distributed system/cloud, you no longer know where is an IP address located.

It’s ingenious.

They said they don’t like stateful NAT, which is understandable. But the load balancer has to be stateful still to perform the routing correctly. It would be an interesting follow up blog post talking about how they coordinate port/data movements (moving a port from server A to server B), as it’s state management (not very different from moving data in a distributed system again).

I am genuinely curious about what’s the appeal of SF?

Managing machine life cycles, data intensive and stateful systems is very hard. I was hoping to see more concrete examples/data points instead of just saying we can also run the same software on our own machines.

In this sense, TV and smart phone waste people much of their lives. And it’s only going to get worse for the coming generations.

It’s very cool.

On the surface, it appears to be similar to Alibaba's HTAP system (alibabacloud.com/blog/what-is-t…). I would love to learn more about how they manage the fundamental tradeoffs between read latency/cost and (essentially unbounded) write amplification/cost. Any paper suggestions?

I will not say cache invalidation is easy; and I am not trying to minimize anyone’s struggles. But cache invalidation is _not_ like FLP impossibility or CAP. Too many systems reason caching (inherently a distributed system) in an ad-hoc way that leads to failures and this belief that cache invalidation is uniquely hard (https://twitter.com/marcjbrooker/status/1534944338341310470?...).

https://en.wikipedia.org/wiki/Consensus_(computer_science)

https://en.wikipedia.org/wiki/CAP_theorem

https://blog.the-pans.com/when-and-how-to-invalidate-cache/ Does this include the "actual cache consistency/invalidation strategy" you were referring to?

I basically wrote a separate blog just for replying to this https://blog.the-pans.com/when-and-how-to-invalidate-cache/.

Good catch! I shouldn't have omitted the details here. Roughly there are two ways to solve the race you mentioned here.

You can use a versioning scheme supported by the database to do compare-and-swap – i.e. sending invalidate along with a hybrid-logical-clock. HLC is nice in this case as it handles DB transaction rollback gracefully, if the database supports it.

Or we can always do the invalidation asynchronously (by recording the invalidation keys transactionally and have a tailer that sends out the invalidation).

I will make an edit to the blog to make it more clear. Let me know if that makes sense!

I mean this – cache invalidate/fill race specifically – is very much a solved problem from a protocol perspective, as long as our definition of the problem is the same – do not leave stale data in cache indefinitely. That is not to say it is easy; I am not trying to minimize anyone's struggles. https://research.facebook.com/publications/scaling-memcache-... might be of interest.

A lot of the challenges in cache are in making the tradeoffs between consistency and coordination overhead based on the workload and the requirement, _and actually_ making caches consistent in production. As explained in the post, there are practical challenges that are very unique to cache and cache invalidation.

More details can be found at https://blog.the-pans.com/when-and-how-to-invalidate-cache/

I wrote a separate blog post focusing on when and how to invalidate cache and how it can be done – https://blog.the-pans.com/when-and-how-to-invalidate-cache/.