aboodman

sync engines only really "work" when you remove the offline part

I don't see the gotcha here. I don't care about the offline part. I mean I accept that some people do, but that's not where the value comes from for many major synced products like Linear, Notion, Superhuman.

complicated/over engineered cache

Sync engines are nothing at all like a cache - that's the point.

They are a replica. Caches are by nature inconsistent. Every entry in the cache is from a different point in time.

Sync engines replicate a consistent subset of your database to the client as an atomic unit. This enables things caches can't do:

  * New, fresh queries can be returned instantly from local state
  * Mutations can apply locally (optimistically) automatically, without custom code
  * The UI updates automatically to reflect server changes

Whether sync engines generalize is an open question. Is it hard? Yes. Is it a distributed systems problem? Yes! Is it worth doing? I think it is. Web applications often suck and sync engines are an important part of many of the ones that don't. I want to enable that experience for more apps without teams having to build it themselves.

What you are describing is exactly what sync engines do. You can have replicas on the server or replicas on the clients. The tradeoffs are the same except the client-based replicas can be in memory, accessed synchronously directly on the ui thread. No server latency at all.

I’m not. See: https://news.ycombinator.com/item?id=48440209

These systems are designed for the very common case of a global user base. If you have geographically centralized users, you can maybe do something simpler. That is rare in my experience - basically all of our customers have users worldwide. They typically don’t even know where their users are so making ux tradeoffs based on that feels really risky.

But maybe my experience is different than yours. One of the amazing things about the software ecosystem is how big it is. Everyone thinks their view is the common case.

Yeah, it's a big part of the tradeoff for this kind of architecture. In Zero, this is a first-class concern via the ConnectionStatus API:

https://zero.rocicorp.dev/docs/connection

Errors and connection are handled in a centralized place so that they automatically get applied to all paths.

Errors immediately disconnect the app and trigger UI. Writes are no longer accepted. After 1 minute of of failed connection attempts, same happens.

This formalizes and enforces the common pattern in popular sync-based apps of detecting disconnects fairly aggressively and warning the user.

We do support Replicache: https://i.imgur.com/R1pR58i.png.

But I get it. Unfortunately something like this cannot be a sidecar, or at least I do not know how to make it one. It's central to your app in the way that React is.

Fortunately the category is expanding and there are several sync engines that plug in exactly how Zero does - by replicating your database. You can switch between them easily. So as long as you think you do need a sync engine you aren't that married to Zero specifically.

Here's a demo of that!

https://youtu.be/SNAHZZo21To?si=wgDgxQpbRr-qj-A-&t=1571

Only in the case where there is such a majority of the company that is tightly geolocated.

Again, AWS latency us-west-1 to us-east-1 is 70ms. That's absolute best case for one round-trip that does absolutely no work. And it's ignoring the case of anyone outside of continental US.

Add in actual server-side work, db interactions, and contention - and you're quickly looking at hundreds of ms.

yeah we specifically decided not to be local/offline-first because these add huge complexity that is not needed for the type of apps we want to support. I spoke about this a bit here if you are interested: https://www.youtube.com/watch?v=86NmEerklTs&t=1764s

As for ZQL:

a) basically all of our customers already use Drizzle/Prisma. So they are very used to custom DSLs, and like them. I know, I was surprised to!

b) You typically use the same code client-side and server-side. There's no branching. The example you pasted is showing an escape hatch for when you want to use custom SQL. The option is there, but it's not the common experience.

This is what a typical mutator looks like:

  ```ts
  // src/mutators.ts
  import {defineMutators, defineMutator} from '@rocicorp/zero'
  import {z} from 'zod'

  export const mutators = defineMutators({
    updateIssue: defineMutator(
      z.object({
        id: z.string(),
        title: z.string()
      }),
      async ({tx, args: {id, title}}) => {
        if (title.length > 100) {
          throw new Error(`Title is too long`)
        }
        await tx.mutate.issue.update({
          id,
          title
        })
      }
    )
  })

We are trying to make apps like Notion, Linear, Superhuman easier to create. These apps all uses custom-built sync engines that took their teams many person-years of effort to construct.

Whether this complexity is worth it depends on how badly you want instantaneous response. If you do, you will end up using sync one way or other, and you will end up with something roughly like Zero mutators.

But this is kind of meaningless unless the tenants themselves are in one geo. Take linear as an example, this strategy works as long as your company that uses linear is all colocated in one area. As soon as you have remote people it falls apart.

If you're interested in this kind of experience for your application, check out Zero (https://zero.rocicorp.dev/).

Live demo: https://gigabugs.rocicorp.dev/.

We also list some alternatives here: https://zero.rocicorp.dev/docs/when-to-use#alternatives.

If you're interested in how these things work internally, check out the Replicache design doc: https://doc.replicache.dev/concepts/how-it-works. Replicache was the predecessor to Zero and the core protocol still works the same way.

Well that "make a mutation client-side" phrase is doing a lot of work.

Make a mutation to what?

The classic server rendered web-app doesn't have any data to make a mutation to. You could try to patch the UI but that would be a huge pita and not really a scalable (in effort) solution.

If you have an SPA, you still don't really have data on the client-side. You have a bunch of cached query responses. You can update those, but (a) it will be a pita to do correctly, (b) you'll have to do it to every possibly affected query, and (c) you have to remember to undo it at the right time (way more subtle than it appears - think it through!).

A sync engine creates the client-side normalized datastore that allows you to "do a mutation client side". In fact, you're kind of right that once you have a sync engine, just doing a mutation is really easy. The real challenge is all the infra required to enable you to do so.

it’s very possible to run a web app backend within ~10ms RTT of most users and have the backend render responses within ~10ms too.

What are you talking about? The only AWS region < 10ms away from us-east-1 is, err, us-east-2:

https://www.cloudping.co/

us-west-1 is 60ms away. eu-centra-1 is 100ms away. asia is 200ms away.

and this is datacenter-to-datacenter traffic. Actual latency over the public internet to residential providers is far worse.

Your database needs to be in exactly one region. So no matter where you put it, the majority of uses on earth are going to be > 100ms away from it.

It doesn't matter where the endpoints are, because the endpoints need to talk to the database to read and write data. Thinking you can replicate some data closer to the users? Congrats you are now the proud owner of a "local-first syncing" database. This replicated database you use (either of your own design or off the shelf) will have all the same problems of client-side syncing. Except you'll still have significant network latency.

There is no getting around physics. You either have quarter-second commits for most users or eventual consistency (aka syncing). Those are the options.

Linus was my boss at Google for nearly 10 years. His main contribution was one of the key people behind Chrome. He's as good as they come.

I was there 2004-2014 and never used an IDE the entire time. From my perspective the most popular editors were emacs and vim. Life was probably different in the Android and Java areas, but there was also a massive chunk (50%+?) of people writing C++ and Python, and I think IDE-less is/was the standard for those folks.

But how can you know who to promote, how to balance resources, or who to hire if you're not leading the project?

People management is about managing the company's resources to achieve goals. If you are not the one leading the implementation of those goals, you are not going to be able to:

  * reason about what the right about of resources should be
  * see opportunities for optimization
  * forecast future need

Do you people only interact with your manager via 1:1? I was constantly interacting with my boss - design meetings, code reviews, product decisions, whiteboard sessions, in slack, in irc ... he was always around.

I got to know him much better through these productive interactions then awkward smalltalk in a 1:1.

And it kind of make sense to meet privately quarterly since perf reviews are also quarterly and that's the only reason I can really think of for a private scheduled face-to-face.

Of course I could always just ask for a private meeting anytime I wanted, which I guess I did from time to time. But it always for a product reason: a tough tech choice I was wrestling with or similar.

I mean a branching factor of 50 vs a branching factor of 7 is a massive difference. A team of 50 can either be run by one manager and a two-level tree or like 8 managers (!!) and a three-level tree. Think about the difference in execution (and expense) in these two companies.

If you can do it w/ the first model why on earth would you not?

The most common split I'm aware of is tech lead / eng mgr. The eng mgr does "people stuff" like hiring/firing and cross-org negotiation, and tech lead does "technical stuff".

But the thing is this makes no sense. Tech issues always turn into people issues - when there is a disagreement, who adjudicates? How can a manager adjudicate something they don't understand. And how will engineers respect / follow the decision?

And people issues invariably become tech issues. How can you hire the right people if you don't understand the tech? How will you know when to fire?

This setup makes no sense to me and i have very rarely seen it work. It seems like it was a product of an earlier time when there was a lot of money floating around and provided a way to (a) shield senior eng from dealing with people problem they just didn't want to, and (b) provide cushy jobs to professional managers that didn't know much about the tech.

But it doesn't work. There's no way to do the shielding well and a person with hiring/firing power needs to know what the fuck is going on.

Really good eng leaders must be both good at tech and good at people. That's the job.

I mean I have 7 reports right now and we're a startup. And fully remote. And I'm still contributing as an IC too.

lol I read "as many as 15+ direct reports" and thought it was hilariously low. My manager at google had like 50+ directs in 2010. And he was the best boss I've ever had.

Popular conception of what a manager is is wildly unambitious.

Weekly 1:1 is performative and useless. It's not what makes a good manager. What makes a good manager is:

  * Having excellent domain knowledge and judgement
  * Having the respect of the team, to settle disputes
  * Solving problems when needed
  * Hiring and retaining an excellent team
  * Picking the right things to work on

If a manager is doing these things well I don't need a standing meeting at all. Or we can meet quarterly to check in.

Email is a thing.

Congrats, Instant team. Genuinely happy for y'all.

But that is a kind of silly way to compare. Broccoli isn't very filling _and_ it doesn't have very much protein in it. That doesn't change the fact that it lack protein.

The question is if I'm preparing a meal that I want to be filling, healthy, and energizing, how should I do it. Broccoli isn't a good answer to the protein part of that question.

I live in Hawaii where the Toyota Tacoma is basically the state mascot. I owned almost exclusively sedans before I moved here. A friend from the bay area moved here several years ago and argued he wouldn't need a pickup, but ended up getting a Rivian within a year or so.

The pickup is more practical than a van here because you end up hauling a lot of dirty/sandy/wet stuff. Yes, you could put this in your van, but hooray now you have sand and water in your van that you need to clean out (and you do need to because the heat will turn it into mildew immediately if you don't). The bed of the truck is outside. It dries out on its own. The sand falls out on its own.

I can't speak for other parts of the US, but use cases can be subtle and I would be slightly cautious about deciding that 300M people and a several trillion dollar market has been completely irrational for decades.

OTOH, I always thought just simplifying to "Timeline" was the obvious move?

Matt, I am so impressed by how far this has come. WRT naming, what about "Journey".

The terminology in this area is still in flux, but for past year or so I've been trying to draw a distinction:

https://i.imgur.com/B5iOd9y.png

https://x.com/search?q=aboodman%20local-first%20zero&src=typ...

It's not so cut and dry.

The majority of the cost in a database is often serializing/deserializing data. By using IDB from JS, we delegate that to the browser's highly optimized native code. The data goes from JS vals to binary serialization in one hop.

If we were to use OPFS, we would instead have to do that marshaling ourselves in JS. JS is much slower that native code, so the resulting impl would probably be a lot slower.

We could attempt to move that code into Rust/C++ via WASM, but then we have a different problem: we have to marshal between JS types and native types first, before writing to OPFS. So there are now two hops: JS -> C++ -> OPFS.

We have actually explored this in a previous version of Replicache and it was much slower. The marshalling btwn JS and WASM killed it. That's why Replicache has the design it does.

I don't personally think we can do this well until WASM and JS can share objects directly, without copies.

Zero isn't trying to be 'truly local-first'.

We're leveraging sync to make high-quality online software. We may come back to offline in the future, but it's not the priority today.

You can read more about our approach to offline here:

https://zero.rocicorp.dev/docs/offline

Big fan!

Replicache, which author loves, is also built on top of IndexedDB.

But notably, not directly atop. We build our own KV store that uses IDB just as block storage. So I sort of agree w/ you.

But if we were to build atop OPFS we'd also just be using it for block storage. So I'm not sure it's a win? It will be interesting to explore.

Hi replicache/zero guy here

Zero (and I believe Replicache as well) layer their own SQL-like semantics on top of an arbitrary KV store, much like the layering of SQLite-over-IndexedDB discussed

Replicache exposes only a kv interface. Zero does expose a SQL-like interface.

I believe they are storing binary byte pages in the underlying KV store and each page contains data for one-or-more Replicache/Zero records.

The pages are JSON values not binary encoded, but that's an impl detail. At a big picture, you're right that both Replicache and Zero aggregate many values into pages that are stored in IDB (or SQLite in React Native).

On the subject of "keep whole thing in memory", this is what Zero does for its instant performance, and why they suggest limiting your working set / data desired at app boot to ~40MB, although I can't find a reference for this. Zero is smart though and will pick the 40MB for you though. Hopefully Zero folks come by and corrects me if I'm wrong.

Replicache and Zero are a bit different here. Replicache keeps only up to 64MB in memory. It uses an LRU cache to manage this. The rest is paged in and out of IDB.

This ended up being a really big perf cliff because bigger applications would thrash against this limit.

In Zero, we just keep the entire client datastore in memory. Basically we use IDB/SQLite as a backup/restore target. We don't page in and out of it.

This might sound worse, but the difference is Zero's query-driven sync. Queries automatically fallback to the server and sync. So the whole model is different. You don't sync everything, you just sync what you need. From some upcoming docs:

https://i.imgur.com/y91qFrx.png