jodah

This is excellent news if it can actually be enforced.

Author here - yes, Failsafe-go definitely took some inspiration from Polly, but it also borrows ideas from the Failsafe JVM library [1]. Polly and Failsafe influenced each other back and forth over the years.

https://failsafe.dev/

Author here - I do get this critique, and of course API design is subjective, but there was a rationale behind the design choices. Putting each policy in its own package allowed the API to be slightly more concise. And having the API use a generic to represent a function's result type allowed different parts of the API to be independent and composable while still ensuring that result types align. A bit more on that here:

https://failsafe-go.dev/type-safety/

There are some (imperfect) workarounds, ex: https://github.com/jhalterman/typetools

I would love to learn how Bob approached interviewing, if you've ever up to writing this up!

While I didn't know Bob personally, he did he an impact on me.

Bob was the first person I looked up to as an open source engineer early in my career. I studied his code closely, and learned a lot from the way he was able to distill complex problems into beautifully crafted APIs. Those learnings, along with some of his principles of software design, shaped my own work, including in open source, to this day. When Bob went on to great success I was not surprised in the least. He was a great engineer and deserved to still be with us.

Atomix has a Jepsen test suite[1], which while not perfect, has helped fix the bugs that existed in the Raft implementation.

1: https://github.com/atomix/atomix-jepsen

"For ILock, the lock is not mutually exclusive under split brain. However it can still be useful as an advisory lock."

Why use a lock that isn't safe when there are other alternatives?

For anyone wondering if there is a set of Hazelcast like data structures and primitives that actually have safe, strong consistency (based on Raft consensus), check out Atomix:

http://atomix.io/atomix/

12B sounds like an incredible number for labs. Is there a source?

Good example of where random retry delays would be valuable. I filed this as a feature to add for the next release:

https://github.com/jhalterman/failsafe/issues/39

Is there a more detailed comparison?

There's nothing more detailed that I know of. Is there a particular feature area/comparison you're curious about? I can add a bit more detail.

It's not apparent to me what the advantage of either interface is. In both situations I have to define a "lambda"

What I meant by this bit is that the user experience is different. Failsafe can be used with method references or lambda expressions [1], which are a nice, concise way of wrapping executable logic with some failure handling strategy. You cannot do this with Hystrix since all logic must be wrapped in a HystrixCommand impl, which cannot be implemented as a lambda.

either seems acceptable.

Like anything, it just depends on what you want. If retries and general purpose failure handling, consider Failsafe. If request collapsing, thread pool management and monitoring, consider Hystrix.

[1] https://github.com/jhalterman/failsafe#synchronous-retries

Good question. Someone asked that recently on Github - here's a quick comparison:

https://github.com/jhalterman/failsafe/wiki/Comparisons#fail...

Author here: fully agree. Blindly retrying an operation any failed operation could lead to cascading failures or system overload, which is what circuit breakers are intended to avoid. Generally, it's just good to think about which failures can or should be retried or recovered from and what recovery should look like. A tool like Failsafe just makes it easier, hopefully, to do what you think is appropriate for the situation.

Author here, Failsafe does support exponential backoffs [1]:

  retryPolicy.withBackoff(1, 30, TimeUnit.SECONDS);

  retryPolicy.withBackoff(1, 30, TimeUnit.SECONDS, 1.5);

[1] http://jodah.net/failsafe/javadoc/net/jodah/failsafe/RetryPo...

[2] http://jodah.net/failsafe/javadoc/net/jodah/failsafe/RetryPo...

Author here - We were aware of Maven Failsafe plugin (since we use it), but felt comfortable using the name anyways since Failsafe is an excellent description of what the library is about, it covers a different technical area than the Maven Failsafe plugin, and "Maven Failsafe plugin" is almost always referred to using that entire name whereas, hopefully, Failsafe will come to refer to this new library.

I have a debit card, I don't need to carry anything else.

If your card hasn't been stolen (via a hacked merchant) and your account wiped out yet, consider yourself lucky. You might want to carry a credit card as your primary instead. When (and I do mean when) it is stolen, at least it's only credit instead of your bank account balance that is impacted (and which can takes weeks or more to get back).

1 CPU still pegged on firefox.

But otherwise, Reason looks very cool. A nice mashup of technologies to create something that certainly looks more useful than the status quo. I hope this takes off, but I think the biggest challenge nowadays, even for good technology, is finding an audience amidst a plethora of choices. Technology, as always, is a popularity contest.

Incredible, but you are correct. Pegs 1 CPU in FF and while it loads, it barely scrolls.

This, usually. It's fair to point out that very specialized clocks, such as Google uses with Spanner, can allow you to achieve some consistency without coordination/locks, but for pretty much anyone else the rule you stated very much applies.

I'm not sure that merely checking the result of some set of operations - whether a mutex is held by the expected process - tells you very much about the validity of how that mutex was obtained (assuming interleaving requests and such). This is where you need to explore the history, which is what the linearizability checker does and which is what Jepsen tests generally use.

It is a bit of work and learning curve writing a Jepsen test suite, but it's not too bad, particularly with the excellent docs that Kyle has recently written:

https://github.com/aphyr/jepsen/blob/master/doc/scaffolding....

is certainly not sufficient to assume correctness of an implementation

Indeed. This is where tools like Jepsen and good fuzz testing can help.

It's related.

Leader election generally[1] requires consensus among distributed processes, and global locks generally[1] require consensus as well. The benefit of this is that both problems can be solved on top of a common consensus implementation which is what http://atomix.io does.

1: I say generally because you can do fancy things with fancy clocks to avoid running operations through a quorum under certain circumstances, but these carry caveats that preclude them from being reliable enough to use in many use cases.

The article calls on someone(?) to Jepsen test Redlock, which is a good idea, but I think that the author(s) of such systems should release their own Jepsen test suite along with the software they're purporting to be safe (as others have done [1][2][3]). I would suspect that operations on Redlock, when subjected to nemesis that partition, kill and skew-clocks on various nodes, would not be found linearizable.

In general, I don't understand why one would build a system that attempts to approximate consensus without just using one of the proven consensus algorithms. Redlock is not the only one here, there are other systems that do this as well.

[1] https://github.com/atomix/atomix-jepsen

[2] https://www.datastax.com/dev/blog/testing-apache-cassandra-w...

[3] https://foundationdb.com/blog/call-me-maybe-foundationdb-vs-...

It is indeed built on Copycat. We're updating the docs right now ahead of an RC, this week or next.

Like anything, it depends on what you're using it for. I wouldn't put a distributed lock into some massively high volume request path, or where absolute availability is required, but it's perfectly fine for some scenarios.

As Martin points out though, many (most?) distributed lock implementations are or can be broken in various ways. He hints at one of the fundamental problems - consensus. Many distributed lock implementations fail simply because they cannot achieve reliable consensus, or don't even try to.

That said, distributed locks can be safe and handle reasonably high throughput. The Atomix distributed lock is one example:

http://atomix.io/atomix/

http://atomix.io/atomix/docs/coordination/#distributedlock

Since consensus requires quorum and quorum requires availability, there is a risk that your ability to obtain a lock or learn about lock related events could be effected by availability, but at least in the case of Atomix, the system is fairly resilient with auto-failovers to passive or inactive nodes as needed (as compared to, say, a ZooKeeper based lock).

On the contrary - I'm glad to hear that he'll remain safe, hopefully, from here on out.

People tend to assume that everyone else thinks like them. Because the arresting officer couldn't wouldn't build a clock for the enjoyment of learning (as opposed to some "broader purpose" like blowing up a building), then nobody else should either. Simple explanation, but likely true.

True, though stupidity is not seeing how much this is going to cost them, ultimately.

Not sure why the current markup couldn't support some responsive styling.