wizeman

but typical compiler behaviors on UB are reasonably understood that I think its immediate effects are overstated

I don't dispute that usually the compiler and the compiled code behave like we expect. That is not under dispute.

What I am arguing is that sometimes, they don't behave like we expect. Instead, they behave in wildly different, unexpected, and unintuitive ways.

Since sometimes they don't behave like we expect, there is no way of knowing for sure what the effects of UB are without looking at the compiled binary code.

So should we say that such code is also vulnerable or exploitable? It would be quite a stretch IMHO.

Sure, but in the case you are describing, the vulnerability is not in the safe code, because if the rest of the code was safe, then the safe code would not exacerbate anything.

In the OP's case, this is not true. The rest of curl could be 100% safe and yet, the bug described in the article could still be the cause of an exploitable security vulnerability.

And since the bug causes UB, there is no way to know for sure if the bug causes an exploitable security vulnerability or not just by looking at curl's source code, unlike what Daniel seems to be claiming.

I expect Daniel Stenberg to actually write something akin to my comments when he dispute the CVE.

Sure. He should dispute the CVE and there is no reason to believe that the bug that he is describing has a security vulnerability.

But once again, his claim that the bug does not cause a security vulnerability is unsubstantiated.

The correct thing to say would be something like: "at this point, there is no reason to believe that the bug causes a security vulnerability".

He could even say that it's extremely unlikely for there to be one. But he cannot say for sure that there isn't one, just by looking at the source code.

The claims he is making contribute to proliferate misconceptions about the C language, C compilers and the security of C code, which unfortunately are too common.

[1] E.g. https://marc.info/?l=llvm-dev&m=143589591927876&w=4

Thanks for the link!

I had never heard about Annex L.

I haven't read the Annex yet, but I suspect that actually achieving the goal of bounding the effects of UB is much, much harder than what it appears to be at first sight.

Right now, it is difficult or impossible to reason about the effects of UB in all cases because the tiniest assumption about the impossibility of a signed integer overflow can lead to an unbounded number of arbitrary, cascaded side effects.

Furthermore, I don't think this is something that can be easily fixed. The main reason for that is that exploiting these assumptions allows compilers to perform significant performance optimizations, and these performance optimizations are correlated with substantial transformations in the IR / compiled code, which sometimes leave the final compiled code unrecognizable compared to the source code.

If you'd prevent the compiler from performing just a single one of these optimizations, many significant real-world software projects and companies would immediately object to that, because it would have a significant performance impact in many important real-world code segments.

I'm talking about projects where a 5% performance degradation in some code segments can be considered a very significant regression.

In the end, the main way to stop UB from affecting compiled code in unexpected ways (in some cases, in extreme ways) is to actually define what happens on an integer overflow, or at least, leave it implementation-defined.

You cannot easily say "you can do whatever you want on integer overflows" while also saying "except the program may not crash just because of the overflow".

The latter is almost equivalent to saying "signed integer overflows must perform two's complement arithmetic", which is what Rust does.

Because any other behavior would basically be just as unintuitive as UB in some cases, and in fact, for you to preserve the existing optimizations, in the worst case the compiled code would have to behave differently depending on not only the surrounding source code, but also any arbitrary piece of code in the entire program (due to link-time optimizations).

The email you linked to even alludes to the Annex's naivety in terms of bounding the effects of UB:

the Annex completely fails to be useful for the purpose you intend

That really doesn't get you much of anything.

But really, all of this is completely beside the point. I was only trying to dispute a couple of Daniel's claims, which are too strongly-worded, without actually disagreeing with anything else that Daniel is saying.

I wouldn't even disagree with those claims if they were worded a bit less strongly.

Integer overflows themselves are not significant, they have to be paired with other mechanisms to be actually vulnerable

No, that is not true. That's a misconception. Your sentence is true for unsigned integer overflows, but not for signed integer overflows, which are undefined behavior in C.

Compilers assume that signed overflows are not possible. They exploit that assumption to perform many significant and important code optimizations, which greatly improve performance. Those optimizations can cause the program behavior to change dramatically in the case that a signed overflow would happen.

In some cases, the compiled code behaves in ways that have nothing to do with how the program was written.

In extreme (but real, documented) cases it can have effects like:

1. Security vulnerabilities being introduced in the compiled code, even though the source code appears to be perfectly safe.

2. Code that isn't called anywhere to be called.

3. The program crashing even though the code looks completely safe, and in fact would have been perfectly safe if the optimizations hadn't been performed.

4. And many more.

These are not theoretical or hypothetical effects. They are real effects that have been demonstrated in real code compiled with widely used compilers (GCC and clang), in some cases in high-profile projects like the Linux kernel, glibc and OpenSSL.

and Rust also agrees, whether overflow checks are enabled or not does not affect the memory safety

That's because in Rust, signed integer overflows are well-defined.

In C, they are not well-defined. They are undefined behavior.

Unless reporters have figured out an ingenious way to exploit a single integer overflow, the OP's reaction is completely justified.

I agree! It is completely justified.

However, the OP's assertions that the code is perfectly safe because it looks like a harmless integer overflow and that anyone can take a look at the code and figure out that it's not a vulnerability, are not justified.

It would be OK to say that the bug is unlikely to be a vulnerability. But not that it is not a vulnerability, at least, not without a much deeper look into why it is not (which would likely include looking at the disassembled code under multiple platforms, multiple compilers and multiple compiler versions).

To be clear, I believe that the burden of proving whether the bug is (or is not) a vulnerability is not on the OP. I would say it would be on those who have assigned a high severity to the CVE.

But that still doesn't make the assertions written by the OP correct.

The practical effect of this security bug is quite limited.

You are probably correct (although it's hard to say for sure that you've considered every single compiler optimization).

But even if you are correct, the point of the parent poster still stands:

something that triggers undefined behavior in C is not to be dismissed as lightly as the article is doing.

I agree with this.

Notice how long your explanation was. The author of the article did not provide a similar explanation, or even an abbreviated one, in order to justify the claims that the bug does not have security implications. In fact, the author didn't even have to provide such an explanation, but he could have at least acknowledged the possibility, which could then be argued to be dismissable based on his (extensive) experience.

I'm not saying that there are security implications or that the burden of proving that there aren't any is on the author of the article.

I'm only agreeing with the parent poster that you cannot easily or simply claim that the bug is not a security vulnerability, without giving some consideration to the possible impact of UB, or at least acknowledging the possibility.

Not acknowledging this possible impact can in fact reinforce misconceptions about the security implications of bugs in C code that lead to UB.

needed more time to solve challenging tasks but made fewer errors [1]

As someone else said, I can solve problems very quickly if the solutions don't have to be correct...

But more seriously, it seems like different people might have different thresholds for when they consider a problem solved?

If you decide to go back and review the problem and solution (even just mentally) to make sure you didn't make mistakes, of course that would take longer and give you more correct answers than the person who doesn't do that?

[1] https://www.bihealth.org/en/notices/intelligent-brains-take-...

being a talented software engineer doesn't give you any ability to understand the nuances of neurochemistry.

This is a bit of a non-sequitur based on what you said before.

Being competent or a talented software engineer is correlated with being an intelligent person, which in turn is correlated with the ability to understand the nuances of [insert topic here].

That doesn't mean that being a talented software engineer necessarily gives you the ability to understand the nuances of neurochemistry, but it is suggestive that it does, if the engineer has an interest in neurochemistry.

Also, being an intelligent person doesn't necessarily mean that you are competent or have expertise in a certain area, sure, but that is not a "logical fallacy" as the GP was saying, nor has much to do with intelligence as he seemed to be arguing (or at least, "fluid intelligence", as it is often understood).

If he simply meant "being competent or an expert in a certain area" doesn't automatically mean that you are competent or an expert in another area, then sure, I agree, and it's true that many comments on Hacker News exhibit this pattern, and this observation is often justified.

But let's not also conveniently ignore the opposite pattern, where a (presumably intelligent) person that doesn't work in a certain field can also provide much better insights than many people with many years of expertise in that same field (especially in things like, say, interpreting scientific papers, which can be legitimately hard to do correctly, for many reasons).

I don't know if you've ever noticed this, but a significant percentage of people of almost every field, even with many years of expertise, is pretty damn incompetent... you just have to step outside your bubble if you don't think so.

This includes software engineering, BTW. There are many software engineers with years of experience who don't even know what a bug tracking system or what a source code management tool are (although, not all of them are incompetent, strictly speaking). That should tell you something.

There's no such thing as "intelligence in one area", intelligence is a general ability. You're basically ignoring everything that psychology has studied regarding intelligence.

That's probably one of the reasons why you'd use GPT-4 to explain GPT-2.

Of course, if you were trying to use GPT-4 to explain GPT-4 then I think the Gödel incompleteness theorem would be more relevant, and even then I'm not so sure.

The Stargate SG-1 TV series has a similar, although not exactly identical, recurrent plot point (replicators). The show is well worth watching IMHO.

You probably don't even need to have the server in your premises.

Perhaps the server could be in some datacenter and you, as well as others, could use it remotely via the Internet.

Didn't Bitcoin lose half its value like 6 months ago?

Let's say the following statement is true: "If X is a scam then X's price eventually goes down". That doesn't necessarily make the converse statement true: "If X's price eventually goes down then X is a scam". This is a common logical fallacy. Those two statements are logically distinct propositions and if one is true, it doesn't necessarily make the other one true as well. This is just basic logic.

Which means that bitcoin's price going down does not necessarily mean that bitcoin is a scam.

Commodities' prices can fluctuate substantially. In October 2008, oil had lost more than 50% of its value in 3 months. [1]

In August 1999, gold had lost more than 70% of its value since its all-time-high in 1980. [2]

Would you also have said at those times that oil and gold are a scam because their value had gone down significantly?

(Cue the "but oil and gold are useful" responses, obliviating the fact that cryptocurrencies such as bitcoin are extremely useful in many ways, some of them unique, and that gold's industrial/ornamental utility does not justify its historical or current price by a long shot, but rather, its utility as an investment/store of value/wealth protection/diversification asset does).

[1] https://www.nytimes.com/2008/10/17/business/worldbusiness/17...

[2] https://goldprice.org/gold-price-history.html

At the very least, I would want to assume any civilization capable of interstellar travel to be at least 1000 years ahead of us, technologically.

My guess is that we'll be able to develop human-level artificial intelligence a lot sooner than 1000 years from now.

At which point it would be a lot easier to send AI probes to explore the galaxy and/or make first contact with another civilization than to build a generation ship.

We'll never know the weather again...

5. Rightfully mentions that you should not put a plaintext password on the CLI, then proceeds to recommend you use pass in a way that puts the password back in plaintext on the CLI via a subshell which is still totally plaintext in the proc filesystem, in ps, etc, to all processes on the system running as any user.

Not to mention that the password will also get stored in plaintext in the shell's history file if the shell is in interactive mode.

If someone replaces the APK on their website with a compromised one, what prevents them from replacing the PGP signature with a signature created by the attacker?

It's very rare and extremely interesting for me to read a well-argued opinion that is the polar opposite of mine.

Furthermore, I don't even have any good counter-arguments (at least not any that wouldn't just be pure speculation).

You definitely gave me something new to think about.

Thank you for your thoughts!

The Flying Spaghetti Monster is also an interesting concept.

It can even be useful as a concept, in certain discussions/contexts.

Should we take the Flying Spaghetti Monster as a core axiom of our theories of physics?

Wouldn't this lead to us logically concluding for example, that there has to exist something else beyond the universe that we know about, even if this is not really true?

I mean, I know that I'm exaggerating and that the analogy is not necessarily very good.

But I'm also not entirely sure how different is the Flying Spaghetti Monster from the infinite objects that mathematicians talk about and that lead us to logically arrive at certain conclusions (that I would argue might not really be true, in terms of things we can understand).

I'm not saying that I'm right and that most mathematicians are wrong, necessarily. Perhaps it's just a linguistic issue, I don't know.

The biggest problem with finite sets is that most operations don't work nicely in finite sets.

I didn't necessarily mean to suggest that we should not work with infinite sets, only that perhaps we should not call them "sets" (so as to not confuse them with the finite ones), but rather something else, like "domains" or whatever.

So perhaps you could say that those commutative/associative operations are valid over the domain of natural numbers. But you couldn't say, "X" represents the cardinality of the set of natural numbers, because there's no such thing as the set of natural numbers.

Or, maybe the set of infinite natural numbers and infinite rationals makes sense but the set of real numbers doesn't (as they have different infinite properties?).

I don't know exactly what would be a better solution here, as I'm not a mathematician. I'm only trying to suggest that there might be a better solution than what we have right now, even if it would simply amount to just using different language and not necessarily changing the logical properties of the theory, so as to not arrive at confusing conclusions.

But it is - you still can't, in practice, construct a useful program that depends on solving the "finite halting problem", since you will need too much memory/time for anything but the most trivial of examples.

But the decidability of the Halting problem is not actually a question of how long it takes for an algorithm to solve it, is it?

The decidability of a problem is a matter of whether it's actually possible at all to construct an algorithm that solves it, no matter how long the algorithm takes to solve it.

That is, whether this algorithm is computable and finishes in a finite amount of steps.

So I will say again that yes, the Halting problem is decidable, as long as you don't use an absurd model to answer the question (i.e. a model that does not represent anything that can possibly be built in our universe even in theory?).

The tortoise and hare algorithm solves the Halting problem for FSMs in a finite amount of time and it only needs twice the amount of storage as the underlying FSM that is being analyzed.

If you don't beleive this, than it should be doable for you to create a program that checks whether P=NP if we limit it to programs that can run on, say, an 8086 processor and only use 16 KB of RAM, right?

The P=NP is a problem regarding the resources required during computation to solve a given problem, as far as I understand.

I am not entirely sure how exactly this relates to the decidability of the Halting problem when applied to finite-state machines, though.

I mean, it's clear that the currently-known algorithms that solve the Halting problem on FSMs, such as the tortoise and hare algorithm, in the worst case take a time to solve that is proportional to the cycle length of non-halting FSMs.

But also note that these currently-known cycle detection algorithms don't even try to analyze the FSM to decide whether it halts, except by testing their states for equality. This is due to the inherent definition of the cycle detection problem.

So they can definitely be made more efficient, perhaps for a large set of useful FSMs, even, as long as they would be allowed to inspect the FSM's state transition table. I'm not sure how efficient they could be, in theory.

I would be cautious in taking any computational complexity theory results at face value when we know that they use Turing machines as models, which can give you results that are the opposite of the results that apply to FSMs. But I'm not an expert and admittedly, I'm not entirely sure of what I'm talking about here.

Regardless, even if you can trust the computational complexity results, to me this is an entirely different question than the decidability of the Halting problem, which is a "yes/no" problem.

That's very interesting, thanks.

So you're saying that in such a scenario, it would be impossible for complex life to exist, even if it looks different from the complex life we have now?

I am just wondering whether we are not confusing "complex life as we've seen to exist" with "complex life that can possibly exist".

For example, I am wondering whether the Great Oxidation Event is not a counterexample where we had a run-away biological process that caused a mass extinction (which, if it were possible for us to be alive at the time, might have looked like it would be impossible to recover from) but still lead to complex life existing despite oxygen being so toxic (at least from the point of view of the kind of life that existed at the time).

I know this is not the best example, as at the time complex life didn't exist yet. But I'm wondering if it wouldn't have been possible for complex life to exist in an oxygen-poor environment. Well, in fact some multicellular species that exist today are anaerobic and therefore do thrive without oxygen, right? Perhaps such anaerobic life would have evolved a lot more if our atmosphere had never been oxygenated.

Unfortunately I know nothing about the examples you mentioned, so I can't tell whether they are comparable.

Edit: I just read a bit about the carbonate-silicate cycle and I think I know what you mean now. Yes, if surface temperatures were more extreme it would indeed be quite difficult for complex life to exist (at the surface, at least). After all, some limits to the existence of interesting life must surely exist and temperature does seem like an inherent one.

I can't really give a substantial response to your first argument, as I'm not certain about all the implications that you mention.

Your second argument, though, doesn't sound very convincing to me.

There are many possible consistent logical frameworks (perhaps an infinite amount of them? hah!) that are clearly not adequate for describing our universe. Or at least, they wouldn't make sense for us.

For a trivial (and perhaps stupid) example, consider an alternate universe where we decided to use a logical framework which can represent natural numbers, but where the digit "2" is represented as "9" and the digit "9" is represented as "7", etc. Let's say you have a 1-to-1 mapping from one digit to another, compared to our usual representation.

Or perhaps the mapping is a lot more complicated than that, and not just about digits, but about entire numbers.

Let's say we decided to do that, for no good reason. Or maybe there was a good reason at the time, but since then we have figured out that it doesn't make sense anymore.

Yes, you could do exactly the same math in this logical framework as we do in our usual ones, and no contradictions would arise.

However, would that really be a wise idea? Wouldn't that just lead to making unnecessary mistakes, sometimes even conceptual ones, when considering that we would be prone to confusing numbers in this system with the numbers that we use normally?

I think language is also important here, not just logical soundness. Especially if we want our intuitions to be helpful.

Just to bring this back to something less abstract again, I will just mention the following:

However, the core of the argument remains true, we just need to complicate it significantly if we want to remain within the realm of finite machines: instead of saying "there is no TM that can decide if any arbitrary TM halts", we need to say something like "there is no finite TM that can decide if any TM smaller than itself halts in an amount of time less than that TM".

Well, I don't believe the core of the Halting problem to be true, and that's exactly the language problem that I'm talking about: confusing language leads us to making conceptual mistakes.

For example, time is not necessarily relevant here.

I could argue that what the Halting problem really demonstrates is that to analyze whether a finite-state machine (FSM) halts, you need to use an FSM that can have more state than the machine being analyzed (which is why it stops working if you believe you can have infinite state).

This FSM with more state would always be able to decide whether the other FSM halts in finite time, regardless of whether the other machine halts or whether it never does.

In fact, that's exactly what cycle-detection algorithms do, such as the tortoise and hare algorithm.

Perhaps we should start admitting to ourselves that infinity is not really something that actually exists but instead it's just an imaginary concept that we created but that has no correspondence to anything that makes sense in our universe?

Or at least, some versions of infinity (I will leave which ones, exactly, to the experts).

Hence all the paradoxes that derive from it.

I mean, perhaps we can work with some version(s) of infinity, without creating a plethora of paradoxes.

The natural numbers, for example, might be something that is infinite and makes sense. However, perhaps talking about a single set that contains an infinite amount of natural numbers might not make sense (or perhaps it still might?). Or perhaps we just need a new language for talking about infinite things without re-using the language we use for describing (finite) sets, so that we don't try to reuse concepts from finite objects with infinite objects.

Or perhaps the only objects that exist (and can be meaningfully talked about) are the computable ones. And/or perhaps "computable function" is one that can use infinite amounts of time but only finite amounts of storage (rather than infinite amounts of storage).

Would it make sense to go back to classical finitism and re-evaluate some of the choices that were made along the way to where we are now?

I can't help but think about the many paradoxes that arise from the very (supposed) existence of infinity.

For example, everybody thinks the halting problem is undecidable. The halting problem is clearly not undecidable on finite-state machines (i.e. real computers), it's only undecidable if you use imaginary Turing machines (which supposedly can have literally infinite state) as a model. But of course, Turing machines cannot exist in our universe.

So it's rather ironic that people say the Halting problem is undecidable, when this is only true for (impossible to construct) imaginary machines but it's not true for the real machines that we actually have and that are possible to construct.

Or think about Hilbert's paradox of the Grand Hotel.

Or the Banach-Tarski paradox.

Or Galileo's paradox.

Or other such paradoxes that only exist if you believe that imaginary infinite objects could possibly exist.

Most people will tell you "well, it's just that our intuition doesn't work for infinite things", but in this specific instance, it's really hard for me to believe that it's our intuitions that are mistaken rather than that we actually made some conceptual mistakes along the way. To me, this argument sounds more like gaslighting rather than a real convincing argument.

I'm not saying our intuition is always correct, though. For example, the quantum properties of our universe are clearly not intuitive but it's hard to argue that our universe is not quantum, after all the scientific experiments that have clearly verified this to be true.

I don't think that invalidates my argument, though, because the convincing argument in that case is not "sorry, you must believe my theory that the universe is quantum" despite not existing any evidence that this was true, but rather, "sorry, but here's all this real evidence that our universe truly is quantum".

Am I crazy here?

You have to shield your home from your neighbor's emitters too. And from the police radar surveillance van sitting in the street in front of your house.

That's what tin foil houses are for, obviously.

But the question was not whether we would be here.

What's to say that in the absence of the K-T event, some dominant dinosaur species wouldn't have eventually evolved their intelligence to our level? Perhaps even earlier?

It's not like the K-T event is a prerequisite for "general" intelligence, is it?

Perhaps in such an alternative universe where the K-T event didn't happen you'd be a dinosaur posting on Dinohacker News right now.

there's a lot that had to go right with the Earth for it to support not just life, but complex, high-energy life, which enables (expensive-in-energy-terms) intelligence

Like what?

It's way more than just being the right size, the right distance from a not-too-dangerous star, and having water.

Besides that and having carbon (and nitrogen, I suppose), perhaps some small amounts of other elements, what are you referring to?

OK, so I have a few comments about your experiments:

I am surprised that mounting worked without error but I guess the device is still active via losetup.

Exactly. `rm` doesn't actually delete the file contents while the file is still open, it just unlinks it from the filesystem tree. So your loopback-mounted disk is still there and all its contents are still available through /dev/loopX.

I'm assuming this would be similar to an actual disk failure though, if the device weren't there maybe btrfs will complain and ask to be mounted with the `-o degraded` flag.

If the /dev/loopX device wasn't there then it would be similar to a complete disk failure, yes.

In this test about 26% of data is still fully readable

It's true that only 26% of data is still fully readable if you account only for files that are fully intact. But also note that about 78% of files were still completely intact.

This is not clear from your comment, but I'm assuming that you are using 4 devices for the btrfs pool as well?

In this scenario, with such a disk configuration and subsequent disk failure you would expect to lose about 25% of files, while the remaining 75% would be intact (especially if the files are small enough)...

But actually, in reality things can be quite better or quite worse, depending on a few factors.

For example:

1. If the free space was fragmented. In such a case, a significant percentage of files might actually be allocated on more than one disk, so you'd lose more files than expected if a single disk fails. Although I can see that on your latter experiment, you've defragged the btrfs filesystem beforehand, so perhaps this is not the main issue.

2. Depending on how btrfs allocates data, if the files are not completely filling all of the disks then they can be heavily skewed towards a subset of the disks.

For example, imagine that each of your disks are 1 TB-sized and your files total less than 1 TB.

In this case, all of your files could be allocated on the first disk only, so losing this disk could lead to losing 100% of your data.

Or for example, if your files are less than 2 TB, they might all be allocated on the first 2 disks only, so losing one of these disks would lead to losing a lot more files than you'd expect if files were evenly distributed across all disks.

But on the other hand, if you'd lose one of the other disks, you might not lose any data whatsoever.

3. Depending on how large files are and how much free space there is on each disk, btrfs might be forced to (or might choose to) span a file across more than 1 disk even on the 'single' profile, even if free space was not fragmented.

4. But of course, more generally, how many files you would lose basically depends on how btrfs allocates disk space across the disks for each file.

These disk space allocation algorithms can be quite more complex than you'd expect from a naive allocator, mostly due to performance reasons.

Unfortunately, I know exactly nothing about how btrfs allocates data, so I can't give you more insight than this, sorry!

At least, that's what you should do if you were performing this test on ZFS,

Just to clarify: in ZFS you shouldn't just 'umount' the filesystem, you should also export the pool, otherwise the cache would still be in use.

What does the 'single' profile do in btrfs?

I thought it was equivalent to RAID-0 mode, but apparently it isn't, because they are separate options and btrfs switched from RAID-0 mode to 'single' mode by default, although the 'mkfs.btrfs' man page doesn't explain what's the difference between them nor why btrfs did that. It has helpful diagrams for the RAID modes, but not 'single' mode.

So how does btrfs allocate data among disks with this profile? This is not explained anywhere I could find. Is it just like concatenating a bunch of disks together?

If so, yes, you would expect corrupting a single disk would completely corrupt a corresponding percentage of the files, but almost nothing on others, assuming the files were 1) written perfectly in order (i.e. were not fragmented) and 2) they were filling all of the disks. In this scenario, completely corrupting one of N disks would correspondingly completely corrupt 1/Nth of files (approximately speaking).

If you were using RAID-0 mode, however, it would be different: it would corrupt almost all (large enough) files, but only 1/Nth of their contents.

The results, however, could vary a lot depending on how file data actually ends up being allocated on disk, of course.

That said, note that the results of this experiment can be completely misleading due to most of the data still being cached in RAM. I would say that to conduct this experiment more meaningfully you should:

1. "umount" the filesystem after having written all data to it.

2. Corrupt the disk.

3. Run "echo 3 > /proc/sys/vm/drop_caches" (a couple of times, just in case). This should get rid of the still-cached blocks in RAM, just in case. Or better yet, just reboot your machine :)

4. Mount the filesystem and check for corruption.

At least, that's what you should do if you were performing this test on ZFS, otherwise corrupting the disks would not affect ZFS's cache and you wouldn't necessarily observe the effects of corruption when accessing the corrupted files normally (i.e. not as part of a scrub).

Admittedly, in btrfs this might not be necessary because btrfs' cached file data might be exactly the same as the underlying disks' cached blocks. This is not true with ZFS, however, as ZFS's caches and the Linux kernel one don't share data (at least, not the vast majority of it).

It would also be helpful to know the distribution of file sizes that were copied into the filesystem, otherwise the "successfully read the entire file" vs "got an error reading the entire file" statistics are not as helpful as it might seem at first sight (e.g. if there is a significant percentage of very small files).

Yes, that's true.

However, even if you're not using ECC RAM, you're much better off using ZFS or btrfs, because due to their frequent checksumming and checksum validations, these filesystems will usually detect memory corruption much sooner than if you didn't use them.

This could be immensely helpful in scenarios such as the great^4-parent poster.

Note that bad hardware is not limited to non-ECC RAM. ZFS and btrfs help just as much in detecting other kinds of bad hardware, such as bad SATA cables, bad disks, bad disk/SATA controllers, bad CPUs, bad power supply, etc.

But of course, once these checksum errors are flagged by ZFS/btrfs, it's a signal to test and fix/replace your hardware, not keep using a machine with bad hardware.

And yes, while ZFS and btrfs cannot fix errors that happen before the checksumming takes place (e.g. due to bad RAM, bad CPUs, etc), they can still detect these kinds of errors (in some cases, at least), especially when they happen after checksumming already took place. And they definitely can detect and fix errors in the rest of the data-to-storage-and-back path (e.g. bad disk cables, bad disks or disk controllers, etc).

I was under the impression running such file systems without ECC was kind of reckless and not a solution either?

It's about as reckless as running any other filesystem without ECC [1].

In fact, you're much more likely to discover memory errors earlier if you use a checksumming filesystem than if you use a non-checksumming one.

[1] https://news.ycombinator.com/item?id=34477035

One of the prime prerequisites of e.g. zfs actually is using ECC RAM.

ECC RAM is not a prerequisite for using ZFS.

Matt Ahrens, co-creator of ZFS and still one of the main developers, said this [1]:

"There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.

I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS."

[1] https://arstechnica.com/civis/threads/ars-walkthrough-using-...

The best way to prove braingenious wrong is by not engaging with him

Well, I like to give someone the benefit of the doubt when discussing these kind of contentious issues (up to a limit, of course). It has served me well in the past (here on HN, for example), and I've learned things that I wouldn't have learned otherwise (from people who hold different opinions than mine).

I also get bothered when I'm accused of something that I'm not, obviously. It's not always easy to move on when someone is saying blatantly false statements about you, as you can imagine.

FWIW I don’t believe you when you say “your language,”

My native language is Portuguese. I was born and raised in Portugal and lived there well until after I was an adult. All of my family and all my childhood friends are Portuguese.

I learned English very early in my childhood, as part of my formal education, due to US cultural exports (i.e. Hollywood movies and TV shows that are broadcast in Portugal without dubbing, i.e. they have always been subtitled and therefore it's much easier to learn the language and become fluent in it) and due to my early childhood interest in programming.

I've always read lots of English prose, especially related to computers and programming (magazines, articles, documentation, websites, Youtube videos, source code, etc, etc). Almost all of this content is from the US, as you can imagine.

I've worked for a few US companies (mostly multinational ones, think FAANG-level) for many years, had many coworkers from all over the world (including many from the US) and always worked remotely, so both oral and written communication has always been in English.

Both due to my professional work and as a tourist, I've spent many months in the U.S., in total (close to a year).

Well into my adulthood, I've also lived for more than a decade in Spain, so I'm also a fluent Spanish speaker.

Also, for many years (perhaps since I started working) I've been reading and listening to way, way more English content than in the other languages I'm a fluent speaker of. This includes Hacker News and other news websites. It includes all media entertainment (Netflix, etc). For decades, I have stopped bothering to find Portuguese or Spanish subtitles anymore for video content due to their mediocre quality or lost meanings, although I do try to put English ones when it's easy to do so.

As another example, I almost exclusively follow English-speaking Twitter accounts (although some Spanish ones as well).

All of this English content is way, way more interesting to me than what I can find in Portuguese or even Spanish. It's not even remotely comparable.

I also don't bother at all with translations from English content which, when they exist at all (well, nowadays there is automated translation, but still), are inevitably mediocre or incomplete due to the much smaller vocabulary in these languages or simply due to the incompetent or error-prone translations.

So to summarize, it would be fair to say that you have culturally captured me :)

It's OK if you don't believe me, but there you have it anyway.