HN user

jdvh

1,705 karma

thymer.com - a local first, end-to-end encrypted task and planning app. Syncs with CRDTs. Optionally self-hosted. Programmable.

Posts19
Comments62
View on HN

We've spend the last couple of years building a local-first end-to-end encrypted multiplayer notes app (Thymer). We also explored running sqlite in the browser but ultimately concluded that we didn't really sqlite or a real database at all in the browser.

What does a document system need? A few simple indexes that track links and backlinks, mentions, and hashtags. You’re broadcasting change notifications anyway (for reactive DOM updates), so updating the indexes for those manually isn’t much extra work. You apply updates eagerly and then notify other users so they can apply the same updates, with some rules to guarantee that all users end up with the same state regardless of the order in which they receive the updates. But a relational database doesn’t help with any of this. Document systems tend to be versioned, so every user action turns into another entry in a transaction log. Even queries like “last Monday’s version of my document” don’t map naturally to SQL. You can query for transactions in a given time period, but unless you have snapshots, you’re still forced to re-apply transactions all the way from t=0 if you want to re-create the document state at a given date.

Hey thanks for mentioning us!

With Thymer we really care about performance, but Thymer is also end-to-end encrypted because we don't want to compromise on privacy. And it's real-time collaborative and offline first.

Thymer has optional self-hosting. Then you can upgrade (or not) at your own leisure, or intentionally stick to an older version you like better. Enshittification is a big problem in our industry. We've all been burned by it -- we certainly have -- and being able to opt out of a "new and improved!" version is a real feature.

Thymer will also be very extensible. Today we launched our plugin SDK: https://thymer.com/plugins and https://github.com/thymerapp/thymer-plugin-sdk/ with a bunch of examples. With Thymer you will be able to "vibe code" the very simple plugins and with VSCode/Cursor you can make more complex plugins with hot-reload.

In theory you can have these "zero cost abstractions" but in practice I don't think so. The user manual gets thicker. Concepts like 'delete permanently' and backup/restore get more complicated. Users will get confronted by scary "advanced users only" warnings in the interface. Some enthusiast blogger or youtuber will create content highlighting those advanced features and then regular users will get themselves in trouble. Customer support gets way more complicated because you always have to consider the possibility that the user has (unknowingly) used these advanced features. If you put buttons in the interface users will press those buttons. That's just a fact of life. Advanced features always come at a cost. Sometimes that cost is worth it, but only sometimes.

Rule of thumb: every 10% increase in complexity cuts your potential user base in half.

This is why people make backups by copy-pasting files. This is why Excel is so dominant. This is why systems like hypercard and git are not mainstream and never will be.

There is a large universe of tools people would love if only they would bother to learn how they worked. If only. Most people will just stick to whatever tools they know.

For most people the ability to go back and forward in time (linear history) is something they grasp immediately. Being able to go back in time and make a copy also requires no explanation. But having a version tree, forking and merging, having to deal multiple timelines and the graphs that represent them -- that's where you lose people.

You're free to argue that this isn't "pure" CRDT, but the CRDT algorithm still runs normally, just a bit later than it otherwise would.

I don't think that's required, unless you definitionally believe otherwise.

When clients disagree about the the order of events and a conflict results then clients can be required to roll back (apply the inverse of each change) to the last point in time where all clients were in agreement about the world state. Then, all clients re-apply all changes in the new now-agreed-upon order. Now all changes have been applied and there is agreement about the world state and the process starts anew.

This way multiple clients can work offline for extended periods of time and then reconcile with other clients.

It's still early, but we have a checkpointing system that works very well for us. And once you have checkpoints you can start dropping inconsequential transactions in between checkpoints, which you're right, can be considered GC. However, checkpointing is desirable anyway otherwise new users have to replay the transaction log from T=0 when they join, and that's impractical.

As long as all clients agree on the order of CRDT operations then cycles are no problem. It's just an invalid transaction that can be dropped. Invalid or contradictory updates can always happen (regardless of sync mechanism) and the resolution is a UX issue. In some cases you might want to inform the user, in other cases the user can choose how to resolve the conflict, in other cases quiet failure is fine.

Completely decentralized is cool, but I think there are two key problems with it.

1) in a decentralized system who is responsible for backups? What happens when you restore from a backup?

2) in a decentralized system who sends push notifications and syncs with mobile devices?

I think that in an age of $5/mo cloud vms and free SSL having a single coordination server has all the advantages and none of the downsides.

I think the main problem is lock-in. If you can't get your data out you can't leave. This is true for open source and for commercial products alike.

If you own your data and if you have the option to self-host you can always opt out of updates you don't like.

Certainly not. I don't believe progress is always good. But subsidies should be reserved for ambitious projects that push the state of the art forward. For those projects that realistically will not get funded commercially. CERN, for instance.

In the short term a free open source govt alternative may be a net positive for society. I don't think it is in the long run. Government projects like these are not likely to really push the state-of-the-art forward. This project even advertises itself as a FOSS Notion alternative. Do government-sponsored clones encourage or stymie innovation? I think the latter.

Every week we read in the news that the EU struggles with entrepreneurship. That our tech industry is languishing. That the EU gets out-competed by the US on software and by China on everything else. Europe should be making industry-leading apps. Europe should produce software startups that make products that get used worldwide. EU subsidized clones of popular American products feels like admitting defeat.

I'm obviously biased because I'm also working on a product in this space. But if Notion developers must become farmers because innovation no longer pays that is a loss to the world in my book.

Thymer[1] uses CRDTs for everything. It's an IDE for tasks and planning. It's a multiplayer app, end-to-end encrypted and offline first, optionally self-hosted, and an entire workspace is a single graph. So CRDTs were the logical choice.

All operations in Thymer get reduced to a handful of CRDT transformations. It doesn't matter whether you are moving or copying text, changing "frontmatter" attributes, dragging cards, uploading files, or adding tags. It's all done with the same handful of CRDT operations. Although this was a lot of extra work up front (we're not using any libraries) the benefits make it totally worth it. When your application state is a single graph you can move text between pages, link between pages (with backlinks), have transclusions, and do all sorts of cool stuff without having to worry about synchronization. CRDTs guarantee that all clients converge to the same state. And because CRDTs are by their nature append-only you get point-in-time versioning for free! We did end up having to make a couple of compromises for performance, though. Version history is not available offline (too much data) and in some cases we resort to last-writer-wins conflict resolution. On balance I think CRDTs are very much worth it, especially if you design an app with CRDTs in mind from day one. I probably wouldn't use CRDTs if I had to retrofit multiplayer in a more conventional AJAX app. Mutations in CRDTs are first applied optimistically, and then when the authoritative sequence of events is determined all clients need to revert their state to the last shared state and then re-apply all events in the correct order (thereby guaranteeing that all clients end up in the same state). Sometimes your app might need to revert and re-apply days worth of changes if you've been offline for a while. This all happens behind the scenes and the user doesn't know how many tree transformations are happening in the background but I guess my point is that CRDTs affect the design of the entire application. Most apps that are popular today were designed back when CRDT transformations were not yet well understood.

[1] https://thymer.com (almost ready for beta)

Both "everything in the cloud" and "everything local" have their obvious technical advantages, and I think they are mostly well understood. What really drives the swing of the pendulum are the business incentives.

Is the goal to sell mainframes? Then tell customers than thin clients powered by a mainframe allow for easy collaboration, centralized backups and administration, and lower total cost of ownership.

Do you want recurring SaaS revenue? Then tell customers that they don't want the hassle of maintaining a complicated server architecture, that security updates mean servers need constant maintenance, and that integrating with many 3rd party SaaS apps makes cloud hosting the logical choice.

We're currently working on an Local First (and E2EE) app that syncs with CRDTs. The server has been reduced to a single go executable that more or less broadcasts the mutation messages to the different clients when they come online. The tech is very cool and it's what we think makes the most sense for the user. But what we've also realized is that by architecting our software like this we have torpedoed our business model. Nobody is going to pay $25 per user per seat per month when it's obvious that the app runs locally and not that much is happening on the server side.

Local First, Forever is good for the user. Open data formats are good for the user. Being able to self-host is good for the user. But I suspect it will be very difficult to make software like this profitably. Adobe's stock went 20x after they adopted a per seat subscription model. This Local First trend, if it is here to stay (and I hope it will be) might destroy a lot of SaaS business models.

Thymer has real-time multiplayer with CRDT based synchronization. With org-mode you can collaborate, but you'd have to DIY with git or something similar. Thymer is also end-to-end encrypted. With org-mode you'd have to self-host something to sync with your phone.

Org-mode is text based, Thymer has a document graph. Advantages of the graph approach is you can nest as much as you want and zoom in on any part of the graph. You can transclude sections and have references and backlinks. With text based approaches you have to sprinkle location identifiers (guids or relative paths typically) into your document so you can track the location as files gets edited. But guids are awkward and paths can result in broken (back)links. We built Thymer as a document graph from the ground up. Tree mutations don't break (back)links.

With Thymer every node in the graph has metadata. Who last edited it? When? Where did it move from? Previous versions? For text paragraphs this isn't super useful, but when you have tasks this metadata is very useful.

org-mode on a phone? I'm sure there are apps for that, but I think it's designed with a keyboard in mind. Thymer uses the (subjectively) more modern command palette approach where you don't have to remember as many shortcuts, and it also translates more easily to touchscreen devices.

For the table situation we solve this by have a generic concept of a "card" that is rendered as a table cell when it's the grandchild of a table (table > tr > td) but otherwise it's just a standalone item, much like a markdown callout.

We've looked at Yjs but ultimately decided to write our own thing from scratch.

Even scale-ups with thousands of users will have people working on different parts of the document tree in practice. The client doesn't need to receive every keystroke for people editing somewhere far away in the document tree. Those updates can be sent in batches every once in a while.

If 10.000 people decide to edit the exact same location in a document then performance will degrade. The client will start to lag behind if it can't keep up with the stream of updates (cpu or internet bottleneck) but eventually it will process everything and all clients will end up with the same state. We have two channels. One for state updates (soft real-time) and one for UI hints ("hard" real-time) and other user actions that aren't CRDT mutations.

We're using a single end-to-end encrypted document tree synced with CRDTs for our collaborative task IDE[1]. All data for a team is a single tree (graph really, if you count transclusions) and its kind of magical how simple everything gets when you know all state will sync in a deterministic way between all clients. It doesn't matter whether you drag&drop some object or add a new user, or rename something. It all reduces to a handful of CRDT operations.

(We have a central server and the app works offline so the algorithm from the linked article doesn't apply in our case.)

[1] https://thymer.com

Oberon (2009) 4 years ago

That's not why Pascal compiles quickly. Resolving undefined symbols isn't difficult or slow, it just means that you have to keep track of what's resolved and what's unresolved and that takes up memory, which was very precious back in the old days of Pascal. Pascal is designed for very fast single-pass compilation, but symbol resolution is only a small part of it.

I suspect this might be why most open source software struggles to reach the quality of commercial products. If it takes 10 attempts to get the major design questions right a company will either get there eventually or get outcompeted. Open source projects typically grow out of somebody scratching their own itch, and that doesn’t involve the scrapping and iterating necessary to make a product that fits many users’ needs.

I know I'm biased because I'm working on an E2EE todo/planning app[1], but over the years I've become convinced that E2EE apps are the future.

All the syncing code you have to write anyway if you want your app to work offline. Despite all the buzz about big data, most individuals and most companies generate only a modest amount of data. Even when you have a long tail of archived data (e.g. emails that go back 10 years) the key benefit is in having all recent data with you, available offline, and end-to-end encrypted.

Self-hosting is a lot more difficult in practice than in theory. Somebody has to make backups, maintain a server architecture with redundancy, apply security patches, etc. For hobby purposes you can get away with just not doing any of that, but for businesses and people who don't want to tinker with their own servers outsourcing all this makes a ton of sense.

I don't agree with your claims that traffic is expensive. In any case, downloading data once and syncing subsequent deltas is still a lot more data efficient than roundtripping JSON for every user action.

[1] https://thymer.com

The problem for (1) is not the camera. It's very likely moisture in one of the connectors in the wiring harness in the tailgate. The fix is contact spray and taping up the connector with duck-tape. (You can test if this is the problem by opening and closing the trunk when you’re in reverse to see if that makes the video worse/better)

It’s also a multistep process with a touch screen: 1) find the button on the screen 2) lean over to reach it 3) touch it and look at it while doing so to confirm you did it right.

In a clickwheel car the wheel moves the focus rect. You twist it blindly to approximately the right spot. Then you look at the screen and adjust one or two clicks and press to confirm. You won’t trigger the wrong action by accident and the focus rect makes the operation async: you don’t have to look while you turn the wheel, you can look at the screen when it suits you.