_urga

"How can we be sure that our radar sensor does not trigger an unintentional break due to a plastic bag blowing in the wind? We collect 10000h of real world driving evidence."

10,000 hours of real world driving data is probably several orders of magnitude too little? Would that really give you confidence that a system can handle a plastic bag blowing in the wind?

I would have thought that this specific scenario would also depend more on factors such as: likelihood of the object being a person or animal vs inanimate, estimated cost of impact to the object and car, and secondary consequences of a rapid deceleration of the car.

A few ideas:

1. Add "abstract" to your search query to surface papers.

2. Search for "... reading list". For example, Heidi Howard maintains https://github.com/heidihoward/distributed-consensus-reading...

3. Read blogs like "The Morning Paper" (https://blog.acolyer.org) but skip fields that are outside your scope. You don't have time to follow more than one or two (or three) major fields.

4. Use Google Scholar to find the most cited papers, or to find papers that build on papers you think are good.

5. Keep an eye out for the conferences where these papers were presented. Then read the other papers that were also presented.

6. When you come across an amazing paper, read other papers by the same authors or supervisors.

7. If you're lucky you might also find good "survey" papers that cover and reference the state of the art.

8. Lecture notes from Stanford or MIT or another university can also be a great way to get a big picture of the evolution of techniques for a given data structure or problem. For example, these lecture notes are just brilliant for getting started with stuff around memory hierarchies: https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_d...

These are a few tricks that I find useful. What else?

Here are some ways to learn deeply and be relevant:

1. Pick a data structure (such as a hash table or LSM-Tree) then read all the literature there is to read, every single paper that's great, following the best conferences year after year, and implement a 10x faster or more scalable version for the std lib of your favorite language.

2. Pick a fault model (such as storage faults, network faults, cryptography faults) then read all the literature there is to read, every single paper that's great, following the best conferences year after year, and write a fault injection or fuzzing harness to break some of the most respected storage/network/cryptography systems (for examples, see the work done by Remzi and Andrea Arpaci-Dusseau on storage faults, Kyle Kingsbury on Jepsen, and Guido Vranken on Cryptofuzz: https://github.com/guidovranken/cryptofuzz).

3. Pick a software field (such as web applications, mobile applications, native applications, file formats such as Office Open XML, or protocols such as SMTP, MIME, HTTP, QUIC) then read as many CVE reports and bug bounty reports as you can find, and then start participating in bug bounty programs within this field. Pick a target and give yourself a goal, e.g. DoS, RCE or read/write access, and do the work to make it happen. Chain as many steps as you can. Automate and enumerate. You'll find a way in if you keep at it. There's nothing like crafting an exploit to change the way you think about programming.

As you gain experience in data structures, storage/networking/cryptography, and security, you'll find this translates well to most software engineering work. You'll gain a speed/safety/security way of thinking, you'll have fun being curious and learning along the way (and hopefully you'll earn a bounty or two and get some CVEs under your name).

Thanks for the pointer to “A Rose for Ecclesiastes”.

I think this comes down to realizing that everything in life is eventually hard. Provided you're doing something worthwhile [1], when the going gets hard, don't look for easier pastures. They're all difficult. Just keep plowing away. And don't doubt. Where there is no doubt, the wind is your companion.

[1] Ecclesiastes is a pretty great book to help figure out for yourself what exactly this is. It's not for nothing that the book uses the same word for "meaningless" or "empty" or "vanity" or "vapor" or "breath" as its refrain. It's also a beautiful book in itself as literature: https://en.wikipedia.org/wiki/Ecclesiastes#Influence_on_West...

"just cranking out variations of that formula as papers."

You'd be surprised how often variations on a formula are all it takes to destroy security. Security is enumeration at the end of the day. If it takes a paper mill to "count all the ways" then I'm grateful.

And above all, simplify, even the charts. Make the big idea clear.

As someone who loves C, Zig has replaced C for me and I don't plan to go back.

There are just too many benefits, not least developer velocity and readability.

Zig is also very approachable to anyone who is familiar with TypeScript, in a way that C isn't. This means more of your team can get excited and get involved, at least in terms of reading the code.

And with Zig, where you have to spend a little extra maintenance time paying for the language being pre 1.0, that's still nothing compared to the upfront and long-lived costs associated with C. You can develop so much faster in Zig compared to C, that you're ahead even pre 1.0.

In the general sense, no, not yet, of course, but in one sense, Zig is probably more production ready than Rust, for example on embedded systems with obscure architectures where OOM-safety is essential.

Here is a talk on using Zig in exactly this kind of environment in production:

https://www.youtube.com/watch?v=124wdTckHNY

It also depends on your project's timeline. If your project will be stable in a year or two, that might well coincide with Zig's stability program.

If Zig's design goals suit your project then it might be worth investing for the long term and gaining deep understanding of the language as it grows.

No worries, I've made the same mistake and everyone in software security probably makes that mistake at some point.

The malicious actor would simply edit the install script to comment out the checksum verification step, since they already own the script.

Yes, just to add:

Raft is essentially Multi-Paxos under the hood, with some overstrict invariants that can impact availability and latency. If you need a consensus algorithm, then it's worth doing a survey of the literature beyond Raft.

Other Multi-Paxos variants are Viewstamped Replication Revisited, which has deterministic leader election for lower latency in the common case and thus no issue of split votes, and which I personally find easier to understand than Raft, also making less demands of the underlying hardware (i.e. VRR can do in-memory leader election, so you're safe in the event of disk failure, corruption, lost writes, and misdirected writes, which Raft does not deal with).

For all these Multi-Paxos variants, you can also look at Flexible Paxos, which is a simple technique for making leader election quorums slightly more expensive (since leader elections are infrequent) in order to make replication quorums much cheaper (since they are the common case critical path). You can use Flexible Paxos to save 20% in the number of hardware nodes for the same common case failure tolerance. Flexible Paxos is a game changer.

I would say quality is undervalued when it comes to technology. Technology suffers from a plague of bloat, and it's getting worse as high level languages pile on the dependencies and abstractions, all the while justifying the waste with "quality is overrated".

I find it hard to believe that you can ship a high quality product without caring about the quality of the components going into it, or that this won't cost you down the line. It only takes a few "good enoughs" in terms of probability theory to end up with a multitude of poor product tail results.

It's also not enough to get the job done, you need to maintain what you ship. You don't need to choose the absolute best tool, but choosing quality tools pays dividends over time.

I don't think it's a question of "boring" or "proven" vs "latest" or "newer". It's not even an 80/20 split between "use proven technology" and "explore new tools".

What it all comes down to at the end of the day is "quality".

You need to have an eye for quality, you need to be able to recognize good design decisions when you see them, appreciate their worth and weigh their potential. You need to be able to make up your mind for yourself.

If you can spot quality in a language, framework (assuming you even need one) or database, then "boring" or "proven" become meaningless phrases.

Quality transcends age and hype.

When you can spot quality far off, you can paddle out to the swell, be there as it breaks, and enjoy a long quality ride that's not cut short because of irrelevance. You won't need to see thousands of surfers on a wave to know it's a quality break.

With technology, the earlier you can spot quality, the sooner you can start riding the wave and benefitting from everything that comes with that (long term experience with that technology, deep understanding, the opportunity to shape or introduce key design decisions at a critical stage, learning from new ideas that can compound earlier into your own output, new connections and marketing opportunities). That's how it always works. There's no point catching a wave because everyone else is riding it. By then, you probably need to be paddling out for the next big swell.

To be clear, this is more than "skating to where you think the puck will be". This is looking for quality and having an eye for it, and being sure of it when you find it.

Is there any syscall to do the equivalent but from C?

e.g. If you were pre-allocating hash tables for a caching service, it would be convenient if the config could stipulate 80% of the system memory instead of the user having to specify the exact amount.

Another example, Zig landed io_uring in the std lib a month ago: https://github.com/ziglang/zig/pull/6356

I'm also really excited by how you can use io_uring to power everything (fs, networking etc.) with one easy api and a single-threaded event loop: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...

io_uring makes thread-per-core designs so much easier.

Beats me!

Cool! I had a similar empirical experience working on a Cauchy Reed-Solomon encoder awhile back, which is essentially measuring xor speed, but I just couldn't get it past 6 GiB/s per core either, until I guessed I was hitting memory bandwidth limits. Only a few weeks ago I stumbled on the actual formula to work it out!

The page cache is not reliable, and actually does more bad than good, especially in the case of PostgreSQL:

https://www.usenix.org/conference/atc20/presentation/rebello

Sure, I'm still pretty fuzzy on these things, but queueing delay is Little's law: https://en.wikipedia.org/wiki/Little's_law

It means if a system can only do X of something per second, then if you push the system past that, new arriving stuff has to wait on existing work in the queue, and things take longer than if the queue was empty. You can think of it like a traffic jam and it applies to most systems.

For example, our local radio station here in Cape Town loves to talk about "queuing traffic" when they do the 8am traffic report, and I always think of Little's law.

Bufferbloat is another example of queueing delay, e.g. where you fill the buffer of your network router say with a large Gmail attachment upload and spike the network ping times for everyone else sharing the same WiFi.

Here is where I got the per-core bandwidth calculation from: https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_d...

The advertised bandwidth for RAM is not actually what you get per-core, which is what you care about in practice.

If you want to know the upper bound on your per-core RAM bandwidth:

64 bytes (the size of a cache line) * 10 slots (in a CPU core's LFB or line fill buffer) / 100ns (the typical cost of a cache miss) * 1000000 * 1000 (to convert ns to ms to seconds) = 6400000000 bytes per second = 5.96 GiB per second RAM bandwidth per core

There's no escaping that upper bound per core.

Nanosecond RAM latencies don't help much when you're capped by the line fill buffer and queuing delay kicks in spiking your cache miss latencies. You can only fetch 10 lines at a time per core and when you exceed your 5.96 GiB per second budget your access times increase.

If you compare with NVMe SSD throughput plus Direct I/O plus io_uring, around 32 GIB per second and divide that by 10 according to the difference in access latencies, then I think the author is about right on target. The point they are making is valid: it's the same order of magnitude.

"Are sector writes atomic? And if so, for what size sectors?"

For a database, this is not the question you want to be asking. It's a rabbit hole. The right answer (which is not always the answer we want, so we keep asking): no one knows.

It's better design to err on the side of assuming disk sector writes are never atomic. This would be taking the security stance, e.g. assume an attacker is actively trying to break your protocol, and use cryptographic guarantees to prevent or at least detect them.

Kernel developers also can't answer these kinds of hardware questions. There's too much hardware out there.

Disks may give atomicity guarantees and then you discover edge cases where these degrade, or disks may emulate Advanced Format 4096 byte logical sectors but have smaller 512 byte physical sectors underneath. Disks are too complicated (and firmware too buggy) to trust any kind of guarantee. There are too many layers of abstraction.

Instead of trying to figure out the safety of the layer beneath you, assume the worst and bring the design back to using end-to-end guarantees.

If you assume and plan for the worst, then you don't need to ask the question, and you can handle the worst of hardware, without surprises when those guarantees are broken.

In fact, with file systems in general, the situation is even worse than for disks, because of disk corruption (3% per 36 month period per disk) and misdirected writes, which most file systems unfortunately don't handle. You should never trust any metadata provided by the file system about your journal file (e.g. file size) and use that as part of your recovery protocol, because the file system itself is storing this kind of metadata on the same disk you're writing too. Again, something as important as the metadata size of your journal file should be protected by your own end-to-end protocol.

all history is revisionist history

That would be the revisionist view.

But history is also one of the sciences, and with a methodology that predates postmodernism.

"If you flip a fair coin 1,000,000 times and get 1,000,000 heads in a row, the probability of getting heads on the next flip is still 1/2."

Does this square with the law of large numbers? The example is almost loaded to begin with, only claiming "a fair coin" but then going out of the way to show the opposite.

I would give the author the benefit of the doubt, but I would have thought a more accurate definition would be that the average outcome for many coin flips would be heads 50% of the time, given enough flips and assuming a fair coin.

So seeing 1,000,000 heads in a row, after 1,000,000 flips, would make me think the probability of heads the next flip is actually 100%, because this coin obviously is not fair.

Technically, if the coin is known to be fair, then sure...

"false negatives" => "true negatives"

If you can reliably get true negatives, you can clear people to go out.

The way to look at it is as a cheap instant filter, something like a bloom filter, that can protect the more expensive tests that take longer.

Your example assumes there's no hierarchy of available tests, and that this test is the only test there is.

What would really happen is those 6 false positives would be referred for a more accurate test. They might miss a day of school but not a week.

At the same time, your more accurate testing pipeline can now speed up thanks to Little's law. There's dramatically less pressure on the system and less backlog, so you have a second order effect that the more expensive slower tests also become cheaper and quicker.

But even if we gloss over all that, and we're only concerned about false positive rate, then this is still much better than no school at all, as in hard lockdown, which has a 100% false positive rate.

Finally, there's the lives saved because of earlier rapid detection and isolation, with corresponding relief for the health care system, leading to increased quality of care and resources available for more severe cases... and so on and so on.

A bloom filter can do wonders for a system, and if this test works it should do the same.

Chaos engineering is brilliant for improving the resilience, safety and availability of a system, but I'm not sure it would help improve the security of a system to the degree that the author describes, i.e. to prevent the SoftLayer firmware hack.

We need more knowledge, more education, more auditing, more communication, and this across all levels of the stack. If we use abstraction, perhaps we need to recognize that abstraction does not absolve us of the responsibility for auditing the implementation, and that abstraction has not only value but also cost.

To make this easier across so many dimensions, we probably want to simplify our stacks and minimize our dependencies and supply chains far more drastically than we do.

There's also a cost to all this understanding, and it comes down to what the business values. Security or velocity. But perhaps you can get both, especially as you design for simpler systems.

"Another example would be the hack of SoftLayer. Hackers modified the firmware on the BMC from a bare metal host the cloud provider was offering. This shows another mistake in having blinders on and not being conscious of the other layers of the stack and the entire system."

Does anyone know whether dedicated hosting companies such as Hetzner or xneelo reflash the firmware on server boards (and disks and everything with firmware in it) before they rent it out again?

And whether they use more forceful hardware techniques to do the reflashing, and not software to politely ask the firmware to reflash itself?

I would hope they would but I can't find any security policy showing they do, or how.

No, the TCP_NODELAY setsockopt is not exposed by the browser.

If you only control the client then you could make your http/fetch requests as big as possible, i.e. do the Nagle algorithm yourself but at a higher level to avoid the delays, but I don't think there's not much else you can do.

Then again, if you control the server you could just run HTTP/3 to be on QUIC and then you're not on TCP any more, you would also save alot of connection handshake latency and benefit from advances in bufferbloat-sensitive congestion control algorithms much quicker.

To be fair, I would be more concerned about bufferbloat delays on small shared networks than delays due to Nagle's algorithm. Bufferbloat delays can be up to tens of seconds, and you can trigger them disturbingly easily, just have someone else in the office upload a large attachment to an email in Gmail's web client to saturate the router's send buffer, while you watch your ping times.

If your library is already merging buffers and preventing small writes then you would still want to set TCP_NODELAY, to eliminate delays due to send/send/recv patterns where merged buffers are less than the MSS... because you know for sure that you're already doing all you can, and Nagle's algorithm can't help further except introduce delay.