jimis

If you use the default mode MDB_SYNC then it's reliable but can be slow for writes. For max write performance you need MDB_NOSYNC (IIRC that's what the official benchmarks use) but then the whole database can be unrecoverable in case of crash. It has happened to me.

Sqlite in WAL mode will never lose all your data and performance can be configured vs durability by setting pragma synchronous to full or normal.

So each enclosure hosts its own RAIDZ2. Have you tested if it can survive loss of USB connectivity? It can happen because of cable damage or movement, and also because of any failure in the enclosure's electronics.

Each drive shows up as an external USB mass storage device?

What is the systemd equivalent for `service crond stop` and later `service crond start`?

In other words, I want to disable all jobs for some time (for benchmarking) and then bring them back up.

  > The DC-ROMA RISC-V Pad II starts at $149 for the 4GB RAM version and is already available for pre-order on the DeepComputing store. 
  > If you get the 16GB RAM variant, the tab is compatible with the Android Open-Source Project (AOSP) Android 15 operating system.

Who are the phone/tablet manufacturers that are friendliest to rooting or installing open-source Android clones? It's a pain to do that on my Samsung Galaxy devices...

I'm looking to buy a (new) 10'' tablet for that purpose.

It's not the same buying a new product from official supplier, to buying something old and/or used from an unknown entity on eBay.

bloat their io buffer size to several GiB, set the eviction priority to 1

How do you tweak these? I'm aware of dirty_writeback_centisecs and the likes, but you are most likely referring to something different.

after paying more money than it would cost to buy a basic laptop

No need to be imprecise, unless you are trying to push some agenda. The cost is CHF 198, which is around USD 195.

They are only slow to compress. They are as fast to decompress, or even faster sometimes. Source: I tested.

JGit has advanced a lot, and is in use in software handling huge repositories like Gerrit code review system. Java has also had numerous advancements in the past 10 years.

It would be interesting to measure git vs JGit performance today.

It would also be interesting to have JGit developers comment on performance after all this time.

The main complaint still stands: the report provides very few details about the incomes and expenses. Out of almost 100 slides, I only see one, titled "Financial Disclosures". And it does not contain many disclosures.

Their financial report appears to be quite opaque and closed, as opposed to the Linux project and community.

Now that SSDs are ubiquitous, hard disk drive manufacturers should up their game:

  * 5.25'' drive  which is taller (more platters) and wider (more sectors per platter)
  * Slow rotational speed to reduce consumption and vibration
  * SMR again, but label the products accordingly
  * Small (64-128GB) SSD embedded, acting as transparent cache especially for quick response to write commands.
    * Possibility to disable this caching layer with a SATA command.

Defaulting to root UID is not an understandable default.

Systemd requiring polkit is a valid issue.

I find this article hard to read. But I believe its point stands. In Rust:

  - You bubble up errors using `?` operator,
  - then you get a nice error message.
  - However the location of the error is lost, the more complex the program, the harder it is to figure out where "permission denied" for example comes from.

  I had implemented it once in 2 lines of Python. some 25 years ago. If I can shake the rust maybe I'll be able to rederive it. 

Awesome to see you answering here! It's obvious that your little experiment has had the honour to be hammered many times by thousands of people, so I'm curious to hear your experience.

Has it been running on a single node all these times? What are the hardware specs? How many requests per second did it handle? Has it gone down because of the load? What was the bottleneck (network, cpu io)? How did it fail (crash, slowness, fire ;-)? Do you have monitoring data to show? How many twitter-like active users would it be able to serve from that single node?

Thanks, and Blërg! :-)

I was looking for Twitter alternatives and found this one, which is old and stale but looks like a very interesting experiment! Quoting its frontpage:

  But what's wrong with Twitter?

  I'M GLAD YOU ASKED. There are two aspects of Twitter that just bug me as an engineer:

    Ruby on Rails - Using rails to prototype a system is fine — scaling up to a million hits a day with it is just a bad idea. As the service grew, I'm sure it cost them a lot more time than it saved.
    140 characters is not enough - I routinely write sentences longer than 140 characters, so I can't even begin to imagine making a point in such a small space. This textual confinement has led to the rise of URL shorteners, which are breaking the internet. 

  Blërg solves these problems by applying absurd reactionary engineering. Blërg's database backend is a custom C program that handles requests over HTTP and stores data in a very small and efficient indexed log-structured database. The frontend is done entirely in client-side Javascript. A single post can be up to 65535 bytes in length.

  Which is not to say that I believe writing your service in C is the solution to all your problems. Clearly, this approach has just as many hairy problems that will bite you in the ass sooner or later. The best way, as with most things, lies somewhere in the middle of high-level abstraction and ZOMGHARDCORE OPTIMIZATION.

   Blërg is a minimalistic tagged text document database engine that also pretends to be a microblogging system. It is designed to efficiently store small (< 64K) pieces of text in a way that they can be quickly retrieved by record number or by querying for tags embedded in the text. Its native interface is HTTP — Blërg comes as either a standalone HTTP server, or a CGI. Blërg is written in pure C.

Thanks for the link to OPAQUE. It looks indeed much better than SCRAM-SHA, I like how the draft underlines that "clients never disclose their password to the server, even during registration."

Do you know if/when it is considered for implementing in the major browsers? Given that even SHA256 digest implementation is missing and that OPAQUE seems much harder to implement, I wouldn't be surprised if it takes another decade.

Regarding not using passwords at all, unfortunately this is not something I can use as a web user with 100s of logins.

If you use SCRAM-, you're committing your password storage to only* use that method for storing the plaintext.

I consider this an advantage. I know how the password is stored, and if the service wants to update the storage, than it has to ask me to reset my password. If it doesn't ask me so, how can I know that they are up to date?

The entire process is invisible to the user, who gets a security upgrade in the process

or never gets a security upgrade, and he has no idea.

I would personally want to see the opposite: disable PLAIN or LOGIN or HTTP basic auth methods or plaintext HTML password-entry forms, and never ever send a password without a digest challenge-response method, even over TLS. Of course first the browsers would have to implement safer alternatives:

https://bugzilla.mozilla.org/show_bug.cgi?id=472823

https://bugs.chromium.org/p/chromium/issues/detail?id=116047...

Why? Because I can't trust all the 100s of companies and I login to, and their 1000s of employees. I don't want them to choose storing my password securely or not. I want them to be forced to do that. Like I choose "HTTPS everywhere", I want to choose "SCRAM-SHA-512 everywhere" or similar.

It's technology that has to provide me guarantees, and user education must only complement that.

Unfortunately we are far from that. Passwords are flying plaintext-inside-HTTPS, with various risks:

* database storing passwords insecurely

* passwords being stolen on the authenticating server (either because of server being pwned or from internal eyes).

* passwords flying fully in plaintext (no TLS) in the internal network behind the authenticating server.

* self-signed HTTPS MITM attacks on already self-signed-cert websites (especially dangerous on intranet websites inside companies; you'd be surprised how difficult it is to setup proper PKI and internal CAs).

Aeskulap was the best one, I used it successfully a couple of years ago, despite the home page making it look like abandonware.

http://www.nongnu.org/aeskulap/

Unfortunately it seems it's no longer available in Fedora repositories.

(including features like undo)

Can you please elaborate on this?

Don't you have the right to remain silent and not answer?

LMDB is a very good choice for many well-known reasons. I don't need to expand here, the advantages are well documented, and more and more projects are choosing LMDB.

However LMDB does not solve all problems, and can be a bad choice for some, and I couldn't find this documented anywhere. Specifically write-intensive workload. Why?

- LMDB by default provides full ACID semantics, which means that after every key-value write committed, it needs to sync to disk. Apparently if this happens tens of times per second, your system performance will suffer.

- LMDB provides a super-fast asynchronous mode (`MDB_NOSYNC`), and this is the one most often benchmarked. Writes are super-fast with this. But a little known fact is that you lose all of ACID, meaning that a system crash can cause total loss of the database. Only use `MDB_NOSYNC` if your data is expendable.

In short, I would advise against LMDB if you are expecting to have more than a couple of independent writes per second. In this case, consider choosing a database that syncs to disk only occasionally, offering just ACI semantics (without Durability, which means that a system crash can cause loss of only the last seconds of data).

Isn't this from the paper "all filesystems are not created equal"? [1] If you search the tables for "sqlite-wal" you will see that it shows zero vulnerabilities.

[1] https://www.usenix.org/system/files/conference/osdi14/osdi14...

Agreed for the read workloads, I have the same experience. It was designed primarily for an LDAP directory after all, an application that is very read-heavy.

Section 4:

"using a 512GB Samsung 830 SSD and an ext4 partition.

The actual drive characteristics should not matter because the test datasets still fit entirely in RAM and are all using asynchronous writes"

And LMDB beats the crap out of SQLite, in any mode. http://www.lmdb.tech/bench/microbench/

Alright, I went through this page to see why my experience is different. While it mentions that they enabled SQLite's WAL journal, it also mentions that the synchronous writes were performed with PRAGMA synchronous=FULL.

I believe that if they set PRAGMA synchronous=NORMAL, they will get an ACI-reliable database that is way faster than LMDB with write-to-disk workloads.

Most of the measurements on that page are not really useful to me; they do not persist the data (db on tmpfs) or they do not care about reliability, using dangerous settings. Only few measurements are both persisting the data and writing safely, but the SQLite configuration is sub-optimal for them.