keyless_

Ultimately, you have to trust your clients to do the right thing.

The lockable server guarantees that e.g. if multiple /acquire requests come in for the same lock in a short time span, only one request will be successful. You are correct in that, without care, there can be pathological cases where a client may not realize their lease has expired.

They are valid solutions for this use-case; the main drawback is you need to set up and maintain a ZooKeeper cluster. Lockable is intended as a simpler and faster-to-use alternative.

It's really up to the client implementation.

In order to deal with long-running processes, the client Python implementation uses a separate thread for sending periodic heartbeats to the lockable server, which serves to do 2 things:

  1. renew the lease so it doesn't expire which would release the lock
  2. notify the main worker thread in the even the lock has been lost

This can be addressed in a few ways - one way being by adding fencing tokens[0]. However, that requires modifying the underlying resource you are accessing.

[0] https://ebrary.net/64834/computer_science/fencing_tokens

Those are valid points.

The Python client implementation can be improved, for sure. In particular, the pathological case that is difficult to deal with is the one where the heartbeat thread pauses at the worst possible time (the "pathological hang" you mention) and so the main thread doesn't notice its lease has expired.

Lockable is designed as an easy-to-use drop-in locking mechanism which requires minimal changes client-side. The downside is that this puts the onus on the client implementation to fully cover all pathological cases.

I agree that the documentation should be clearer about these limitations and requirements on the client.

Thanks for the feedback, it's really useful!

Unfortunately, it doesn't. That was actually the reason I built this originally - there's no easy way to control read/write access to a file on S3 without using some sort of external locking system.

Conceptually, that's how it works, however, instead of keeping an HTTP connection open, lockable expects to receive periodic heartbeats /hearbeat to keep the lock acquired. The TTL is variable, and can be set depending on the use case.

Good questions:

How do you ensure that a lock isn't acquired by two requests?

Indeed, the compare and set is done atomically. It's guaranteed that the lock can only be acquired by at most one process.

How do you release a lock reliably? How do you solve the problem of releasing accidentally while using a resource?

A lock is released in one of two conditions:

  1. the /release endpoint is called
  2. the lease on the lock expires

Can the lock jam locked if the process dies?

Locks come with a lease which expires after a set amount of time. If lockable doesn't receive a heartbeat to renew the lease, the lock is released automatically.

I would use Consul for this or I would try avoid needing to lock to begin with.

For sure Consul is an alternative, so are ZooKeeper and things like ETCD - lockable is intended to be a no-setup alternative to something like that.

Unfortunately, you do need _some_ sort of TTL because there is no way for the lockable server to know when a remote process has simply died.

However, TTLs can be set to be arbitrarily long (e.g. multiple days), which means, in practice, you can avoid losing locks due to networking issues.

Yes, any process can call a heartbeat. If the lock hadn't been acquired (by any process), the heartbeat call fails.

Lockable itself isn't distributed (or, at the very least, doesn't appear distributed from the outside), so I'm not sure if linearizability applies here, but maybe I'm misunderstanding you comment.

In a similar vein, I guess you can ask what happens if a client first checks if a lock is available then tries to acquire it, in two separate steps; but in that case, there's no guarantee that the lock wasn't acquired in between checking and acquisition.

A long held view of mine is that our current architecture where we keep logic on the server and data in the database is rather unfortunate, because it means you either have to

* marshall data to your server to process (which can be slow), or

* marshall code to your db to run (which can be messy, in part because you're translating from one programming language to your db's language - this is the main reason ORMs exist)

I always thought that the solution is to have all computation live in the DB (similar to how kdb does it), but this is pretty nice too. If you squint your eyes this looks like a much cleverer ORM.

It would be even better if the language supported SQL as a first class citizen, but that's not really an option here, since you're stuck with js.

Indeed, the first version of this heavily borrowed from this approach.

The trickiest part is correctly writing the logic that spawns the secondary thread used for heartbeats.

Probably ease of setting up and not needing to manage a piece of infrastructure - which is the case with all service offerings, really.

It's not a difficult conceptual task to keep track of some locks in a Postgres DB (or use PG advisory locks), but you still need to:

  * make sure all processes can access the db (directly or indirectly)
  * make sure the db can handle all the connections (or set up PGBouncer if you think you're going to be handling many processes at the same time)
  * write some client-side logic to acquire locks, retry on failure etc.