mprime1

Very cool!

Here’s something similar-ish but using symmetric encryption (i.e. password):

https://mprimi.github.io/portable-secret/

https://news.ycombinator.com/item?id=34083366

Shared something very similar on HN a few years ago. The entire source code is about a hundred lines, which makes it easier for folks to review and trust: https://news.ycombinator.com/item?id=34083366

https://mprimi.github.io/portable-secret/

There are some restrictions.

I had the foolish idea of installing a Tesla charger at home to charge my Bolt. I’ve been unable to ever use it.

The wall charger works fine with Teslas. My car and adapter charge fine at Tesla superchargers.

But the home Tesla charger refuses to charge my Bolt. (Yes I disabled vehicle restrictions and tried all sorts of combinations of settings for weeks before giving up. Tesla support was useless of course)

Restriction or bug, same difference.

5.9k stars, 727 forks, 132 contributors: https://github.com/nats-io/nats.go

That's the NATS Go _client_.

The server project is https://github.com/nats-io/nats-server 17k stars, 1.5k forks, 160 contributors

Mastodon is another one (ActivityPub / Feediverse)

https://nlnet.nl/project/Mastodon/

never really understood how just adding extra invisible stuff to make the equations work is justified

IANAP but here’s my understanding.

At the end of the month you spent $2000, you’re not sure how so you track down your expenses:

  - rent $500
  - groceries $120
  - gas $80
  - …
  - unknown: $123

My anecdotal evidence confirms this. Friends in my home country are hooked on Telegram meme groups and as a result they’re spreading [Russian propaganda] conspiracy theories at alarming levels.

(Russia is really good at weaponizing memes but I don’t meant to single it out, the US has also been very successfully influencing the same country for decades via Hollywood movies for example)

Stay humble

  - There is someone else that is better at X than you are
  - The person not so good at X is probably better than you at Y

Same thing as what?

Parent was asking about deduping encrypted data.

Someone said (wrongly) it’s impossible and I shared a popular project that does exactly that.

Indeed. Borg for example is e2e but able to dedupe.

My bookmark archive is 10TB but deduped on-disk size is 100GB because most files are the same across backups!

https://www.borgbackup.org/

I had a fascinating conversation with someone that has been working on this system for the last 20 or so years.

Water treatment is powered by the geyser and in turn the leftover brown water feeds the geyser.

Pretty neat (for sewage)!

Ordering gives you ... ordering. And nothing more.

Process 1 receives: 3:00 [1] "P1 is leader" 3:01 [2] "P2 is leader"

Process 2 receives: 3:00 [1] "P1 is leader" 4:00 [2] "P2 is leader"

This is perfectly valid Atomic Broadcast. Order is maintained.

However from 3:01 to 4:00PM you have 2 leaders (or 2 processes holding the lock).

Don't use ABCast to do locking / leader election for your "user-space" application!

Rounding up 'usually works' to 'is reliable' is a recipe for building crappy systems.

Rounding down 'usually works' to 'it's not perfect and we need to handle edge cases' is how you build dependable systems.

Your first comment seemed very much in the first camp to me.

The fatal flaw is calling them "locks".

When programmers think of locks, they think of something that can be used to guarantee mutual exclusion.

Distributed locks have edge cases where mutual exclusion is violated.

Implementation does not matter.

e.g. imagine someone shows you a design for a perpetual motion machine. You don't need to know the details to know it doesn't work! It would violate the laws of physics!

Similarly, anyone telling you they created an implementation of a distributed lock that is safe, is claiming their system breaks the laws of information theory.

"Distributed locks" are at best contention-reduction mechanisms. i.e. they can keep multiple processes from piling up and slowing each other down.

[Some] Paxos for example use leader election to streamline the protocol and achieve high throughput. But Paxos safety does NOT depend on it. If there are multiple leaders active (which will inevitably happen), the protocol still guarantees its safety properties.

They both reduce to a paxos style atomic broadcast

Atomic Broadcast guarantees order of delivery. It does not (cannot) guarantee timing of delivery. Which is what people want and expect when using distributed lock / leader election.

There's some serious flaws in your reasoning.

TCP guarantees order [as long as the connection is active] but it is far from being 'perfectly reliable'.

Example: sender sends, connection drops, sender has no idea whether the receiver received.

In other words, it works until it doesn't. The fact that sometimes it doesn't means it's not perfect.

TCP is a great tool but it doesn't violate the laws of physics. The 2 generals problems is and will always be impossible.

I'm sorry, I didn't mean to be bashful. I am not familiar with S3 and maybe what you describe is a perfectly safe solution for S3 and certain classes of usage.

I could not get past the point where you promulgate the idea that ZK can be used to implement locks.

Traditionally a 'lock' guarantees mutual exclusion between threads or processes.

"Distributed locks" are not locks at all. They look the same from API perspective, but they have much weaker properties. They cannot be used to guarantee mutual exclusion.

I think any mention of distributed locks / leader election should come with a giant warning: THESE LOCKS ARE NOT AS STRONG AS THE ONES YOU ARE USED TO. Skipping this warning is doing a disservice to your readers.

Notice I did not mention FLP.

Atomic Broadcast (via Paxos or RAFT) does not depend on partial synchrony assumptions to maintain its safety properties.

Your internet or intranet networks are definitely asynchronous and assuming delays are bound is a recipe for building crappy systems that will inevitably fail on you in hard to debug ways.

"Distributed locks" are at best a contention-reduction mechanism. They cannot be used to implement mutual exclusion that is _guaranteed_ to work.

I've seen way too many systems where people assume TCP == perfectly reliable and distributed locks == mutual exclusion. Which of course it's not the case.

Telling people "here's a recipe to do locks" should come with a giant flashing sign that say: "this is not an actual lock (as in in-process locks) -- locks in distributed systems are impossible, this cannot be used as a recipe for mutual exclusion"

Thank you for the link GP!

I’m the ‘maintainer’ but I’m hands off and not planning on significant improvements.

Discussion on HN was also quite interesting and you may find some ideas: https://news.ycombinator.com/item?id=34083366

I also recently presented this at HOPE(.net) and was very well received by a technical crowd so congrats on independently inventing the same thing ;-)

Leader election and distributed locking reduce to the same problem… which is proven to be impossible. It means in some edge case it will fail on you, is your system handling those cases?

I didn’t read past this:

Systems like Apache ZooKeeper or Postgres (via Advisory Locks) provide the required building blocks for this

Zookeeper is the original sin. Convincing a whole generation of programmers that distributed lock are a feasible solution.

This is my biggest pet peeve in distributed systems.

——

And if you don’t believe me, maybe you’ll trust Kyle K. of Jepsen fame:

However, perfect failure detectors are impossible in asynchronous networks.

Links to: https://www.cs.utexas.edu/~lorenzo/corsi/cs380d/papers/p225-...

https://jepsen.io/analyses/datomic-pro-1.0.7075

Definitely!

1. Many companies are hiring "AI" engineers. My guess is 90% of these jobs are virtue signaling to investors and those positions will go to folks that are good at appearing competent in interviews. Yay! More overpaid incompetent colleagues, just what we need.

2. My editor saves me about 2 minutes a day with smart printf/loop/variable completions (JetBrains editors -- no sarcasm, i like this!)

3. I am wasting time responding to email from PMs suggesting that "we don't have engineering capacity to do XYZ, but maybe we can use an AI to do it???"

(I am not anti-GenAI -- I've used it to create flyers and do pretty cool stuff!)

FYI Evolutionary Algorithms have been an active area of research for decades.[1]

Among the many uses, they have been applied to ‘evolving’ neural networks.

Famously a guy whose name I can’t remember used to generate programs and mutations of programs.

My recommendation if you want to get into AI: avoid anything written in the last 10 years and explore some classics from the 70s

[1] https://en.m.wikipedia.org/wiki/Evolutionary_algorithm

The AI training team asked for this feature

(I’m being a bit provocative and assume today it stores locally only but a future TOS change will secretly and “anonymously” upload your data ‘for training purposes’ —- that’s what everyone else is doing these days)

(Author) at the moment I use git-annex for my primary archive.

The upsides are: - it’s git! Everything is versioned and easy to sync to one more place - it uses disk more efficiently than git (no double disk usage for objects and checkout)

The downside: - it’s a pain to get it building and working on a new system (Haskell)

I'm sorry. I should have had a copy-paste warning -- typing from a phone... I knew it was coming.

Simplified version I can type from memory (I.e. on a system where I don’t have my gitconfig aliases):

  -—graph -—oneline -—color

Great read. Great product. I've been an early user of Antithesis. My background is dependability and formal distributed systems.

This thing is magic (or rather, it's indistinguishable from magic ;-)).

If they told me I could test any distributed system without a single line of code change, do things like step-by-step debugging, even rollback time at will, I would not believe it. But Antithesis works as advertised.

It's a game-changer for distributed systems that truly care about dependability.

It often is.

I had my PayPal personal account immediately frozen and closed without recourse just for (unintentionally) putting a few hundred dollars of bitcoin through a mixer.

1. Purchased $200 of Bitcoin via PayPal (never doing this again — took weeks to settle) 2. Created a receiving wallet address with a wallet app 3. Moved Bitcoin from PayPal to new wallet address.

This was innocuous, I wasn’t trying to cover my tracks in any way.

What I didn’t know is that the wallet had an automatic tumbling feature, so the money I put in got mixed.

And PayPal immediately closed my account. Tried every way to get it back but no dice. All funds frozen for 6 months before I could redeem them.

So yes using a mixer can get you tainted.