HN user
https://msun.io
ah, so something like... if the primary ordered transaction 3 < transaction 1, but transaction 2 observes only transaction 1 on the read-only secondary potentially because the secondary orders transaction 1 < transaction 3?
"I think what's going on here is that a read-only transaction against a secondary can observe some transaction T, but also miss transactions which must have logically executed before T."
i was intuitively wondering the same but i'm having trouble reasoning how the post's example with transactions 1, 2, 3, 4 exhibits this behavior. in the example, is transaction 2 the only read-only transaction and therefore the only transaction to read from the read replica? i.e. transactions 1, 3, 4 use the primary and transaction 2 uses the read replica?
Discord | Remote (US) | Senior Software Engineer, Persistence Infrastructure | Full-time
The Persistence Infrastructure team develops and operates Discord’s real-time datastore systems that serve the data of Discord's 150M+ monthly active users—including over a trillion messages! We work across multiple systems areas: databases, disk storage and Rust-based data access services. We're a small team whose work has a huge impact on our organization's success and ability to grow.
Apply here: https://discord.com/jobs/7288234002
I think there's some confusion at what distributed databases like Spanner, Cockroach, etc do:
- A given row of data is configured to have N replicas, e.g. 3, 5, etc, but N doesn't have to equal the actual size the database cluster. e.g. you can have a database cluster of 21 nodes with N being 3. N represents the "Raft cluster size" and the database is composed of many "Raft clusters" that are responsible for segments of the data space. You can horizontally scale the size of the database cluster by adding more nodes and thus more "Raft clusters"
- Raft/Paxos are used to ensure linearizable writes to those replicas through a leader
- Multi row/partition transactions are handled through a different kind consensus called atomic commit/2 phase commit that's orthogonal to Raft/Paxos consensus: https://timilearning.com/posts/mit-6.824/lecture-12-distribu...
- Systems like Spanner, Cockroach have higher write latencies because replicas in Raft cluster have to achieve quorum consensus, but reads latency can mostly not be affected by always reading from the leader of a cluster via things like leader leases.
FDIC only insures up to $250K
Is it really a problem to bubble up that error (and kill the database server) if you can just bring up a new database server with a clean cache (potentially even on the same computer without rebooting it) instantly?
Our estimations for MTTF for our larger clusters would mean there'd be a risk of simultaneous nodes stopping due to bad sector reads. Remediation in that case would basically require cleaning and rewarming the cache, which for large data sets could be on the order of an hour or more, which would mean we'd lose quorum availability during that time.
to me your solution sounds absolutely brutal for needing a complete copy of the remote disk on the local "cache" disk at all times for the RAID array to operate at all, meaning it will be much harder to quickly recover from other hardware failures.) In Scylla/Cassandra, you need to run full repairs that scan over all of the data. Having an LRU cache doesn't work well with this.
Is it necessary to fully replicate the persistent store onto the striped SSD array? I admire such a simple solution, but I wonder if something like an LRU cache would achieve similar speedups while using fewer resources. On the other hand, it could be a small cost to pay for a more consistent and predictable workload.
One of the reasons an LRU cache like dm-cache wasn't feasible was because we had a higher than acceptable bad sector read rate which would cause a cache like dm-cache to bubble up a block device error up to the database. The database would then shut itself down when it encountered an disk-level error.
How does md handle a synchronous write in a heterogenous mirror? Does it wait for both devices to be written? Yes, md waits for both mirrors to be written.
I'm also curious how this solution compares to allocating more ram to the servers, and either letting the database software use this for caching, or even creating a ramdisk and putting that in raid1 with the persistent storage. Since the SSDs are being treated as volatile anyways. I assume it would be prohibitively expensive to replicate the entire persistent store into main memory. Yeah, we're talking many terabytes.
I'd also be interested to know how this compares with replacing the entire persistent disk / SSD system with zfs over a few SSDs (which would also allow snapshoting). Of course it is probably a huge feature to be able to have snapshots be integrated into your cloud... Would love if we could've used ZFS, but Scylla requires XFS.
Scylla (and Cassandra) provides cluster-level replication. Even with only local NVMes, a single node failure with loss of data would be tolerated. But relying on "ephemeral local SSDs" that nodes can lose if any VM is power-cycled adds additional risk that some incident could cause multiple replicas to lose their data.
That's good observation. We've spent a lot of time on our control plane which handles the various RAID1 failure modes, e.g. when a RAID1 degrades due to failed local SSD, we force stop the node so that it doesn't continue to operate as a slow node. Wait for part 2! :)
We're actively hiring for our storage infrastructure team (SF or remote) ! If this sounds interesting, check us out!
Yes. And we're hiring storage engineers! https://discord.com/jobs/5411664002 https://discord.com/jobs/5426301002
byte | New York, San Francisco, Remote (USA) | Full Time | iOS/Android/Golang/OpenGL | https://byte.co
Brought to you by some of the original creators of Vine, byte is an app where creators make short looping videos. Our goals are to democratize creativity and give our audience tools to make creative, interesting, and unique content.
We launched at the end of January and have fundraised enough to get us to our next goal. Our technical stack includes Go, Spanner, and Google Cloud Platform for the backend, and Swift/Kotlin on our iOS/Android apps respectively. We're also exploring the use of OpenGL on our mobile clients.
We're looking for engineers that are thoughtful and excited about helping creators express themselves and building new tools and algorithms to help people browsing the app discover content that suits them.
We offer competitive salary and equity, great benefits, a flexible/remote working environment and fun problems. Come join us!
Android engineers: https://jobs.lever.co/byte-2/8969f10b-e81f-48ff-830a-7c7079f...
iOS engineers: https://jobs.lever.co/byte-2/f4fe0b53-690b-40c4-9c7e-467bdce...
Video graphics engineers: https://jobs.lever.co/byte-2/abab572f-bb53-45a5-8eb2-d112d3c...
Backend engineers: https://jobs.lever.co/byte-2/6eb31ae6-7c44-438c-b406-45cbbc9...
byte | New York, San Francisco, Remote (USA) | Full Time | iOS/Android/Golang/OpenGL | https://byte.co
byte is an app where creators make looping 6 second videos, brought to you by some of the original creators of Vine. Our goals are to democratize creativity and give our audience tools to make creative, interesting, and unique content.
We launched at the end of January and have fundraised enough to get us to our next goal. Our technical stack includes Go, Spanner, and Google Cloud Platform for the backend, and Swift/Kotlin on our iOS/Android apps respectively. We're also exploring the use of OpenGL on our mobile clients.
We're looking for engineers that are thoughtful and excited about helping creators express themselves and building new tools and algorithms to help people browsing the app discover content that suits them.
We offer competitive salary and equity, great benefits, a flexible/remote working environment and fun problems. Come join us!
Android engineers: https://jobs.lever.co/byte-2/8969f10b-e81f-48ff-830a-7c7079f...
Backend engineers: https://jobs.lever.co/byte-2/6eb31ae6-7c44-438c-b406-45cbbc9...
iOS engineers: https://jobs.lever.co/byte-2/f4fe0b53-690b-40c4-9c7e-467bdce...
Video graphics engineers: https://jobs.lever.co/byte-2/abab572f-bb53-45a5-8eb2-d112d3c...
Benchmark ran on each DB’s default isolation mode which are different: serializable for Cockroach and snapshot for YugaByte.
I hope Regulator has been decommissioned ;)
Philosophically, does the hyper.sh approach reflect Exokernel's vision? https://en.wikipedia.org/wiki/Exokernel
Mailbox was acquired by Dropbox some time ago.
somebody tell the governor to read this:
"Brave New Ballot: The Battle to Safeguard Democracy in the Age of Electronic Voting". Aviel Rubin.
happy for this move. i was actually starting to envy microsoft's metro design aesthetic and loathe the changes that came in every new os x release.
uh, where'd you get 24? how about 10 X 9 X 8 X 7 = 5040 combinations.
You're exactly right stickfigure.
* You have to make sure the attributes are properly signed by the OpenID provider which is what the security advisories by Google and OpenID foundation were about.
* You can't trust any OpenID provider to give you a correct and verified email address. For the specific case of Google single sign-on, you can trust the Google OpenID provider.
What if Apple provided some sort of 2-factor auth that you had verify with the phone rep? Like they'll send you an email or sms and you verify the code back to the rep?