One Word Broke C

https://news.quelsolaar.com/2020/03/16/how-one-word-broke-c/

by quelsolaar • 6 years ago

147 251 6 years ago

news.quelsolaar.com

lmm9• 6 years ago

Where is the constituency that would want this "sensible C"? People who care about reasonable behaviour have already moved on to better languages, a la https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporativ... ; the people still using C are those who want "maximum performance" and don't think undefined behaviour is a problem, and so C compiler writers have (understandably) gone ever further in their exploitation of undefined behaviour to get that "maximum performance".

I don't think this can be reversed, and I don't think we should necessarily want to reverse it. We have better alternatives now. Let the C people do their thing, and get on with your life in a language that ensures that programs, even "incorrect" ones, have reasonable behaviour.

lone_haxx0r• 6 years ago

What are these "better languages" you speak of? Because I've been looking for a good C replacement but to no avail. C++, D, Nim, Go, Rust, Zig, Odin, etc; I've looked into them but none convinced me.

admax88q2• 6 years ago

Stop looking for another C. Move on to something really different, something dynamic like JS, Clojure, Python, or Smalltalk

worthless-trash• 6 years ago

If none of the above convinced you, you'd need to specify why each of them didn't convince you (to do what) to get a qualified answer here.

Every design choice has a repercussion, specify what repercusison that you would find acceptable and then your ideal language can be defined.

lmm9• 6 years ago

What were your issues with those languages? Several of them sound like reasonable options to me.

For a mature, general-purpose language my default suggestion is OCaml. If you really absolutely can't make garbage collection work for your case (something I've never seen happen to anyone who actually tried) then your choices are more limited (and I'd probably favour Rust, despite its relative immaturity).

There are any number of good languages out there, and I could happily go into the details of which I think offers the best combination of tradeoffs (Scala). But the bigger picture is that a memory-safe language with the ML featureset (in particular first-class functions, parametric polymorphism, type inference, and sum types) should be the minimum baseline these days, and represents a substantial step up from C (at the most basic and pervasive level, being able to do error handling with a result type vastly improves your defect rate). Within that category you have plenty of reasonable choices offering their own particular selling points.

adrianN• 6 years ago

I'm pretty sure that a large part of the people working on legacy code bases would like to have a safer language. You don't move on to a better language if that means rewriting your whole project.

lmm9• 6 years ago

"Sensible C" would have to be a distinct language from C, at least as the latter is currently implemented, so it would still represent a change of languages for legacy codebases. It could probably interop with the C ABI, but we already have other languages that can do that; it's not at all clear that converting your codebase piecemeal to "sensible C" would be substantially easier than doing the same with, say, Rust.

pjmlp11• 6 years ago

I pretty much doubt it, given that ISO C WG doesn't care one second about making C safer.

It is Oracle with SPARC ADI, Google/ARM/Apple with ARM MTE, Microsoft with Checked C, AT&T with Cyclone that actually bother to change the status quo.

dooglius• 6 years ago

raises hand

The sort of optimizations that compilers abuse UB for aren't that useful, actually, and they don't actually require a compiler anyway; it's more about compiler writers overfitting for benchmarks.

justinmeiners2• 6 years ago

I regularly use other "modern" languages and still enjoy using C. Am I cult member?

lmm9• 6 years ago

Cultishness is a spectrum, not a binary. C has always had a certain proportion of adherents who believed that "performance" (of code that implements microbenchmarks with unlimited hours of manual tuning) is the most important aspect of a general-purpose programming language, and that the problems of undefined behaviour are insignificant. I would say that both those viewpoints are crazy, but they are increasingly becoming the mainstream in the C community, via the mechanism described in the link: more moderate people are driven out, rendering the community more extreme, which then drives more of the moderate folk out, and so on.

dejj• 6 years ago

Quite the converse. To me, “permissible” sounds as capricious as does “possible”. I read the article on “mass amateurization” before and am thinking how words must be sounding differently to me as a non-native English speaker and computer scientist. Maybe it had been best to insert “No running with scissors!” to that spec.

dependenttypes• 6 years ago

You might find this relevant https://news.ycombinator.com/item?id=10772841

What you want is not another language but a compiler that has a well defined behaviour and is not willing to change it between versions when UB in the C standard is triggered.

_kst_2• 6 years ago

The author claims that:

In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”.

That's a serious misinterpretation. C89 says that "Permissible undefined behavior" includes "ignoring the situation completely with unpredictable results". There is absolutely no requirement to document what those "unpredictable results" might be.

The standard joke is that one possible consequence of undefined behavior is making demons fly out of your nose -- not because that's actually possible, but because actually making demons fly out of your nose would not violate the standard. That's equally true in C89 and all later versions of C. The change in C99 from "Permissible" to "Possible" made no difference. The phrase "imposes no requirements" has always meant exactly that. (And if you think the change from "Permissible" to "Posisble" is semantically significant, then it would have been a recognition of what was already accepted.)

The idea of "nasal demons" goes back to 1992. http://catb.org/jargon/html/N/nasal-demons.html

mjd2• 6 years ago

I came here to say something similar. The author says:

In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”.

This is wrong. That is the meaning of what C89 calls “implementation-defined” behavior. The implementation gets a choice; the standard may lay out what the choices are, and in any event the choice must be documented.

But with C89's “undefined” behavior, all bets are off. Numerous examples are documented of cases where real compilers were forced to behave in utterly bizarre ways because of real hardware limitations.

eru13• 6 years ago

The C standard is quite liberal in its use of undefined behaviour.

Some cases are due to conflicting existing implementation that predated the standard. (Though implementation defined might make more sense?) Some are for performance reasons.

And some are utterly bizarre, like some things that should be syntax errors instead being declared undefined. If memory serves right, that includes eg not closing your string literals. (Thankfully, all implementations I know give you an error message.)

paulddraper2• 6 years ago

Yep. The author's mis-statement:

In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”.

Wrong. The entire point of "undefined" was to mean "the standard imposes literally no requirement."

"Implementation-defined" and "unspecified" are essentially what the author is thinking "undefined" means. To quote the standard:

3.4.1 implementation-defined behavior unspecified behavior where each implementation documents how the choice is made
EXAMPLE: propagation of the high-order bit when a signed integer is shifted right.
3.4.3 undefined behavior behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements
EXAMPLE: the behavior on integer overflow.
3.4.4 unspecified behavior use of an unspecified value, or other behavior where this International Standard provides two or more possibilities and imposes no further requirements on which is chosen in any instance
EXAMPLE: the order in which the arguments to a function are evaluated.

quelsolaar10• 6 years ago

"Ignoring the situation" is not the same as do whatever you want.

aw162110719• 6 years ago

And "do whatever you want" isn't really what compilers are doing either. Out of all possible program executions, they are ignoring the situations where undefined behavior is invoked. That (arguably) matches what the standard says, does it not?

TwoBit3• 6 years ago

Yeah, though that's largely irrelevant to the point of this document.

tracker1• 6 years ago

LOL... just crash the app, and then document that it's supposed to make demons fly out of your nose and that it only works with blood sacrifice hardware installed.

"Demon summoning failed, nasal hardware not found."

saagarjha15• 6 years ago

You are looking for a version of C that traps on undefined behavior–which is totally fine!–but do know that this would either be something that does not optimize code and/or contains runtime security checks for things that C traditionally does not have checks for.

gpderetta7• 6 years ago

yes of course, you would need a well grounded summoning grid.

fulafel3• 6 years ago

As a metapoint, the fact that there is this confusing epistemological minefield around the specified semantics is a bad thing in itself, separate from the evils of UB.

jacinabox• 6 years ago

The implementation of stdlib necessarily contains code which invokes implementation defined behavior, and the reason that the compiler implementation doesn't treat it in like manner is simply that it doesn't identify it as such or is prevented from doing so. So the treatment of libraries doesn't need to be consistent with that of user programs. Practices that ensure that such code is not optimized away include scattering the information needed to do so across multiple translation units, so that the compiler does not put them together.

paulddraper2• 6 years ago

You're not wrong but what are the alternatives?

(1) Have a vague spec that doesn't distinguish between defined and undefined behavior.

(2) Define everything, at the cost of performance/hardware support.

Remember that C works on 50-year-old hardware, and current hardware.

MaulingMonkey• 6 years ago

The author claims that, recently, the ocean has been getting concerningly wet. GCC 1.17 dates back to 1988 and would launch nethack as an easter egg in response to something as tame as implementation defined behavior (#pragma use.)

This is incorporated into a gamedev variation on "nasal demons" I've heard/parroted: "It can launch Nethack, it can launch nuclear missiles at a cow ranch in Alaska, or worse - it can do exactly what you expected it to do." The first two being perfectly standards-compliant behaviors when your code has an RCE vulnerability thanks to invoking undefined behavior via, say, buffer overflow. The last turning compiler upgrades into a minefield, and bug trackers into a dumping ground for once-encountered heisenbugs.

3JPLW• 6 years ago

Regardless of the cause (if it's that "one word" or not), the behaviors are crazy-making and there's a trend of more aggressively exploiting UB for optimizations.

kps3• 6 years ago

I've become convinced that the original ANSI committee didn't intend to create the monster they did. The reason is that Dennis Ritchie wrote a submission to the committee¹ that described the ‘noalias’ proposal as “a license for the compiler to undertake aggressive optimizations that are completely legal by the committee's rules, but make hash of apparently safe programs”, and, that “[i]t negates every brave promise X3J11 ever made about codifying existing practices, preserving the existing body of code, and keeping (dare I say it?) ‘the spirit of C.’”

Those comments describe what ‘undefined behaviour’ turned in to. The only reason dmr and others didn't make the same objections to it is that nobody realized at the time what the committee wording implied.

¹ available here: https://www.lysator.liu.se/c/dmr-on-noalias.html

amluto• 6 years ago

I don't buy these arguments. Suppose you have a little bit of UB:

    char foo[10];
    foo[10] = x;

That's a buffer overflow, and a clever attacker can supply input for which this really will format your hard drive. Writing a standard that allows literally anything for this particular example of UB but somehow requires sensible behavior for other types of UB is not obviously possible. Merely saying that it can't happen doesn't make it so.

quelsolaar10• 6 years ago

Thanks for the link!

kazinator5• 6 years ago

The author's interpretation is completely spot on. C89 does not allow the implementation to deliberately break a program that invokes undefined behavior, unless they document what they are doing. It may allow the program to break due to ignoring the situation.

There is absolutely no requirement to document what those "unpredictable results" might be.

Only in that case: when the situation has been ignored completely.

If the situation has not been ignored completely (for instance, the compiler analyzes for the absence of that type of undefined behavior and optimized based on that), then it is not permissible to behave with unpredictable results. The behavior now falls into a "documented manner characteristic of the implementation", or else terminating at translation or execution time with or without the issuance of a diagnostic.

One way to look at this is that they changed "permissible" to "possible" for a reason. What might that reason be? Almost certainly that the word "permissible" gives requirements.

aw162110719• 6 years ago

If the situation has not been ignored completely (for instance, the compiler analyzes for the absence of that type of undefined behavior and optimized based on that)

Except optimizing based on undefined behavior is also a form of ignoring the situation. For example, say you have code that can:

1. Execute Action A if undefined behavior is present, or

2. Execute Action B if undefined behavior is not present.

A compiler could look at these two possibilities, see that one of them invokes undefined behavior, ignore that situation completely, and optimize the remaining path. And that would be perfectly in line with the wording of the standard.

The interpretation you and other commentors who dislike optimizations based on UB favor is that "ignoring the situation" should mean "treat the code as if it was conforming". Both are valid readings of the standard, and the only way one will become the dominant reading would be for the standard to use less ambiguous wording.

kazinator5• 6 years ago

On the other hand, this is arguably just a simple defect in the C89 wording.

In the previous sentence, it is clearly written "... for which the Standard imposes no requirements."

The issue is then that "permissible" appears to give requirements.

That's an apparent contradiction in wording, which they rectified by changing "permissible" to "possible", which is a justifiable action that doesn't change the language.

radisb• 6 years ago

Honest question, since I have no expertise in the field: If C89 says that "Permissible undefined behavior" includes "ignoring the situation completely with unpredictable results" can you give an example of something that would constitute non permissible undefined behaviour according to the standard?

mjevans2• 6 years ago

Any instance of undefined behavior should result in a Warning, if not an Error.

Similarly, 'dead code paths' should also result in a Warning, if not an Error. (Possibly with a way of turning that off for specific functions.)

I also fully agree with the author's statement about what a good compiler should be doing.

millstone• 6 years ago

Any instance of undefined behavior should result in a Warning, if not an Error

How would you implement this? For example, use-after-free is undefined, and difficult to detect.

vnorilo2• 6 years ago

Warning when optimization passes take advantage of UB is a reasonable desire. The compilers are getting increasingly good at this, but I believe it would be far too noisy to blanket enable all of that.

jcelerier2• 6 years ago

C++ is able to do it in constexpr contexts - a constexpr function cannot exhibit UB (else it's a compile error)

pjmlp11• 6 years ago

Most static analyzers are already able to spot a big chunck of such cases, even if not everything, e.g. clang tidy and VC++ 2019 lifetime profile.

Additionally, SPARC ADI, ARM MTE and CHERRY can be used to invalidate memory regions after free and trap on further accesses.

While not perfect, better than nothing.

gpderetta7• 6 years ago

Any instance of undefined behavior should result in a Warning, if not an Error.

Beyond trivial cases, UB is a runtime behaviour, not compile time.

'dead code paths' should also result in a Warning, if not an Error.

the noise generated would be unbearable.

icedchai• 6 years ago

Most of the UB people complain about is related to aggressive, compile time optimizations. This may or may not be trivial. Often it isn't, since decades old "working" code can be broken by new compilers. You can argue that the code was always broken, of course.

sbmassey• 6 years ago

This would presumably include any pointer dereferences where the compiler cannot prove that the pointer is always non null.

bluGill7• 6 years ago

There is good reasons compilers don't warn about most undefined behavior: there is so much might be but really isn't undefined behavior in the world.

I often write code that won't divide by zero, something I can prove because I know my whole program while the compiler only knows the file/library it is working with at the moment. As such I didn't put in a if divisor is zero check, and now want the compiler to waste my time on a warning that is wrong.

throwaway20482• 6 years ago

this is a complete misunderstanding of what undefined behavior is, the compiler doesn't detect UB and then call emit_nasal_daemons().

nneonneo3• 6 years ago

My "favourite" recent compiler bug comes from Rust. Compile the following code in Release mode:

    fn main() { (|| loop {})() }

This defines an anonymous function containing an infinite loop, and calls it. Any sane compiler would make this just loop forever, right?

    $ cargo run --release test
    Illegal instruction (core dumped)

Yeah, that's right. In Rust, an ostensibly safe language, Clang's overeager optimizer chooses to emit an illegal instruction in lieu of an infinite loop. (Try it here: https://play.rust-lang.org/?version=stable&mode=release&edit...)

This particular bug actually comes from C++. In C++, threads may be assumed to always make forward progress, i.e. side-effect-free infinite loops are UB. However, this is not true in C, and it's definitely not true in Rust, yet Clang makes this assumption anyway.

It gets a lot worse; once Clang sees an infinite loop and decides it's UB, it gets to make all sorts of thoroughly invalid and silly "optimizations"; see this SO question for some examples: https://stackoverflow.com/questions/59925618/how-do-i-make-a...

lmm9• 6 years ago

This particular bug actually comes from C++. In C++, threads may be assumed to always make forward progress, i.e. side-effect-free infinite loops are UB. However, this is not true in C

That's very much disputed. The C standard requires programs' behaviour to be as-if executed on the C abstract machine when they terminate. It's not at all clear whether the standard imposes any requirements on programs that would not terminate when executed on the C abstract machine.

_kst_2• 6 years ago

C11 does have a special case rule for infinite loops. Section 6.8.5 paragraph 6:

"An iteration statement whose controlling expression is not a constant expression, that performs no input/output operations, does not access volatile objects, and performs no synchronization or atomic operations in its body, controlling expression, or (in the case of a for statement) its expression-3, may be assumed by the implementation to terminate."

(I'm personally not happy that this is stated in terms of that the implementation may assume rather than what a program may do.)

Note that this does not apply to infinite loops with constant conditions. For example, given `while (1) { /* ... */ }`, the above doesn't apply and the implementation may not assume the loop will terminate.

pingyong6• 6 years ago

Is there even a case where the compiler can optimize a terminating loop better when it is allowed to assume that it will terminate at some point?

unlinked_dll• 6 years ago

That's a shame because you could get an essentially one-instruction program

    loop: 
       jmp loop

CE shows this happens on opt-level=1 which is neat. Of the weird breakages this takes the case!

https://godbolt.org/z/LyERew

nneonneo3• 6 years ago

The posted program is one instruction too!

    main:
        ud2

(ud2 is a permanently illegal instruction on x86)

jfkebwjsbx3• 6 years ago

Rust does not use Clang. You are thinking about LLVM.

But the bug is actually on rustc side because they fork LLVM on purpose and use it as they wish.

If the rustc folks merge their project into LLVM, then you could claim it is an LLVM bug, otherwise please do not.

nneonneo3• 6 years ago

Ah, nomenclature fail on my part. You're right, it's LLVM.

This is an LLVM bug - it affects C code too (when it should only apply to C++). If you compile this code:

    static void die() { while(1) ; }
    int main() { die(); }

as a C program with optimizations, it will die with an illegal instruction too.

Diggsey• 6 years ago

This is rubbish. The bug exists in LLVM regardless of whether Rust has its own fork, and you don't even need to use Rust to reproduce it.

Furthermore, using your system (ie. unmodified) LLVM with Rust is supported, and still exhibits this bug.

bigcheesegs2• 6 years ago

Yet again someone ignoring the line:

for which the Standard imposes no requirements.

C89 had no requirements, C99 clarified that the line that followed was non-normative.

The entire premise of this article is flawed.

quelsolaar10• 6 years ago

The premise is that just because the standard doesn't impose requirements, it doesn't mean that other known factors like ABI, Architecture, and other platform specifics don't matter.

TwoBit3• 6 years ago

The point of the article is that the author wants compilers to tell users when they do unexpected things. I don't see how your point is relevant to that.

bigcheesegs2• 6 years ago

That may be the point, but it's not the premise. Even the title is based on this flawed idea. If the author wants to make a point, they can do so without trying to drive it with incorrect ideas.

clarry2• 6 years ago

And the problem is that users ask the compiler to optimize, and then they complain because it optimizes.

See, optimization can do a lot of unexpected things, but it is very difficult in general for the compiler to tell whether a certain application of (say) range analysis to eliminate dead code in a bunch of inlined calls is going to be unexpected. That's exactly the kind of optimization I want a compiler to do; I can write small functions that are as generic as possible yet cover all the edge cases, and then the compiler can find out which edge cases cannot apply in this particular situation.

If the compiler told the user every time it did something, you'd never finish reading the output. Might as well ask the compiler not to optimize at that point.

gpderetta7• 6 years ago

The compiler doesn't know it is doing unexpected things. The compiler is doing what's expected given the axioms it has to work with. It would be nice if the compiler added assertions in place to validate that the axioms hold, but that would often be very expensive. Recently I have been using the sanitizers extensively and has been a huge improvement, but of course they are too expensive to leave on in prod. Would be nice if a cheaper, although less exhaustive version were available.

userbinator• 6 years ago

This topic comes up semi-regularly here on HN but I think the underlying issue is that standards do not exist in a void, yet a certain group of people seem to think that they do --- the fact that a standard "imposes no requirements" should really be taken to mean "think about what really makes sense", not "do whatever the fuck you want". Hence the "but we still complies with the standard!" defense from compiler authors when faced with perplexing results of UB is totally ignoring the reality and practicality of what a compiler is for. Note that the standard writers have even tried to give that hint, with the "behaving during translation or program execution in a documented manner characteristic of the environment" phrasing which is precisely what programmers are usually expecting, but it seems few actually took the hint.

mpweiher3• 6 years ago

IIRC the C standard explicitly states that conformance to the standard is insufficient for having a compiler that is fit for purpose.

pjmlp11• 6 years ago

The best example are embedded platforms, full with extensions that probably will never be part of ISO C.

giomasce• 6 years ago

Different people have different ideas on what "makes sense". The point of a standard is to not have to use common sense, which is usually not common al all. So there is no point is saying that when the standard does not say anything you should use common sense. If the standard says nothing, you should just avoid relying of that thing.

The compiler is not responsible for the quality of the source code. The programmer is. The compiler is responsible for the quality of the machine code, assuming that the source code is correct. It is good to have tools to check the quality of source code, but they are a different thing from compilers.

Personally, I am happy that the compiler optimizes the code for me when I write correct C/C++ programs. If I make a mistake and inadvertently do UB, I take the responsibility, without shifting it on the innocent compiler.

pjmlp11• 6 years ago

The compiler is not responsible for the quality of the source code. The programmer is.

The last 40 years have proven how well it works in practice, specially if one plugs some kind of networking into it.

Taking the responsibility should be taken to the same liability level of other engineering disciplines, then it would be interesting to see how long the myth of only bad programmers write bad C will survive.

_martamoreno_• 6 years ago

Really bad example... Memset to zero members of a structure this way is actually something that should trigger a compiler error, because it makes no sense. I have limited understanding of C these days, but I consider that a good thing, because it allows me to look with fresh eyes on the crap I wrote back in the days. And this is one of them. It's nonsensical and similar to the general linux sentiment of naming your variable like random garbage. It's brainfuck. The only thing I can say with some distance is that exploiting undefined behavior in pretty much all cases should yield a compiler error.

saagarjha15• 6 years ago

I have limited understanding of C these days, but I consider that a good thing

I wouldn't.

tumult12• 6 years ago

This is shown as an example of illegal C:

    struct{
    char x
    char y;
    }a;

    memset(&a, 0, sizeof(char) * 2);

But the only thing illegal about it is that there's no semicolon after 'char x'.

Writing to (theoretical) padding in this way is fine. Though, you might want to check the manual to see if this could generate a trap representation on your platform. (This won't, on any platform I'm aware of.)

Also, the sizeof operator always returns 1 for char and unsigned char (C99: 6.5.3.4), so there's no reason to do sizeof(char).

hackcasual• 6 years ago

Agreed, that's not undefined behavior. It's just not guaranteed to 0 out y

tumult12• 6 years ago

It actually would be guaranteed to 0 it if it were instead unsigned chars. For plain char, the implementation has to specify if it is signed, and if it is, what kind of padding it has. If it has none, it would also be guaranteed. Therefore, you can just check the compiler manual to see if this guarantees zeroing. (None of the mainstream compilers on mainstream platforms will have padding for signed char, either.) (C99: §6.2.5 and §6.2.6.2)

(If you know a platform where this is not true, I'd be interested in hearing about it!)

Edit: Oh, also, an implementation is allowed to have different padding in struct fields than the padding of the type itself. But it has to define this, so you would be able to look it up in the manual to see if it's different. (§6.7.2.1: "Each non-bit-field member of a structure or union object is aligned in an implementation-defined manner appropriate to its type.")

cygx4• 6 years ago

It's just not guaranteed to 0 out y

Citation needed. As far as I'm aware, memsetting structs is perfectly fine. The problem are things like memcmp as padding bytes take unspecified value.

unwind• 6 years ago

I didn't understand why you would want to write that code. To me, a struct is a (very very mild) abstraction, since there can be padding added for alignment and if you don't need to know about each field's offset, then you shouldn't care.

So just write

    memset(&a, 0, sizeof a);

and be done, that will zero any padding too and end up just doing the right thing. It's also very clear to the compiler what you're after, and I wouldn't be at all surprised if a compiler chose not to call memset() for this, and just does the equivalent of

    a.x = 0;
    a.y = 0;

or perhaps, by knowing about padding, doing a properly-sized single write to both fields at once.

tumult12• 6 years ago

Why is beside the point. I'm not making a judgment about morality or motivation. I said that one of the few concrete examples in the blog post was factually wrong. It shouldn't be used as an example for the point the author is trying to prove.

Also, for proving the point of the blog post, the example you showed instead would have been wrong in the same way as the original example from the blog post.

The point was about writing to the (theoretical) padding between fields. Your example would still have written to this padding (if it existed) in the same way. And if this padding did exist, it still wouldn't have been illegal, in either example.

saagarjha15• 6 years ago

that will zero any padding too and end up just doing the right thing

No, it will not; the compiler can and occasionally will transform this into two writes that don't touch padding. It is surprisingly difficult to actually zero out padding bits. (However, your code is better than the one mentioned above, as it does correctly zero out the structure's members.)

SAI_Peregrinus2• 6 years ago

A more useful example would be

    struct {
        char a;
        char b;
        char c;
    }a;
    
    memset(&a, 0, 2 * sizeof(char));

IE trying to clear only part of a struct using memset. Clearly not a great way to do things.

mark-r5• 6 years ago

Did you miss the part where writing to a padding byte might generate an error on some architectures? Granted I don't know of any such architectures, but C was designed to work on just about any weird thing you can dream up.

aw162110719• 6 years ago

I'm not really convinced the author is correct in claiming that a one-word change opened the floodgates to optimizations on undefined behavior. In particular I think:

Careful reading will reveal that the word “Permissible” has been exchanged to “Possible”. In my opinion this change has lead C to go in a very problematic direction.

is a red herring. In my opinion, the actual problematic phrase is this:

ignoring the situation completely with unpredictable results

which didn't change between C89 and C99.

It all comes down to what "ignoring the situation" should mean. Compiler vendors appear to interpret this to mean "ignore situations that invoke undefined behavior". Programmers who dislike optimizations based on undefined behavior appear to interpret this to mean "ignore the violation that leads to undefined behavior and treat it like conforming code". Who's right? It's ambiguous.

C compilers have taken the concept of undefined behavior even further by doing the mental acrobatics of thinking that “If undefined behavior happens, I can do want I want, So therefor I can assume that it will never happen”.

I think this description is a bit uncharitable. I think a better description might be "If undefined behavior happens, there are no constraints on program behavior. Thus, if I assume that undefined behavior never happens, optimize the program based on that assumption, and that assumption is violated, the resulting program behavior is still OK because the standard imposes no rules on what the program must do when undefined behavior is invoked."

Lets take a look at signed integer overflow:

Wasn't this useful for loop counters? I think this gist had more info: https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759...

In addition, this bit:

In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”.

is implementation-defined behavior, not undefined behavior. As the C89 standard notes, defining the implementation's behavior is only one of the things an implementation may do, not something it must do.

a1369209993• 6 years ago

is implementation-defined behavior, not undefined behavior.

They're (confusingly) misusing the word "define" there. Implementation-defined behaviour is required to be documented, but every behaviour is something that "you must define" in the sense that you have written down some (possibly implicit/emergent) definition of it as part of the compiler's source code. Ie, it's not "the standard requires you to define this", it's "you are not capable of not defining this, and the standard requires(C89)/doesn't require(C99+) you to pick from the following options".

aw162110719• 6 years ago

I think I see what you're trying to get at. It feels almost tautological, but I think it makes sense.

mpweiher3• 6 years ago

What a great idea!

https://blog.metaobject.com/2018/07/a-one-word-change-to-c-s...

The comment section also has a lot of the same justifications for the current interpretations that I see here, with fairly thorough debunkings.

Changing that one word is probably not quite enough, at the very least you need to also not make the remaining text a note or also change the status of notes back again.

Then there is the conflict between "imposes no requirements" and "here is the range of permissible actions". These cannot both be true, and the current dogma is to resolve the conflict by pretending the second part dosen't exist (with, admittedly, some justification). But then why on earth is it in the standard at all?

saagarjha15• 6 years ago

"Ranges from" does not necessarily enumerate the only possible choices: https://english.stackexchange.com/questions/148428/using-thr...

mpweiher3• 6 years ago

Never said it did...

jancsika2• 6 years ago

If the compiler thinks that writing to NULL, is undefined, it can therefore assume that since you are writing to p, p can’t be NULL. If p can’t be NULL, the entire if statement can be removed and after optimization the code looks like this:

Is that correct?

I thought Linus' rant was about NULL checks that came after a dereference had already happened. In that case it at least makes sense that the compiler would assume the NULL check would be superfluous.

But how could branching to spit out an error and exit before the dereference ever get optimized out? I don't see any undefined behavior in the author's example upon which an optimization could trigger. (Or if it could trigger, it ought to trigger on array bounds checking and many other situations where removing the code would clearly cause bugs.)

quelsolaar10• 6 years ago

You get this bug because the compiler doesn't know that the function will not return and therefor assumes that the de-reference will happen even if the value is NULL. Some compilers have a (non standard) keyword to indicate that a function will not return. adding an "else" will "fix" the issue.

aw162110719• 6 years ago

This is a corner of C I'm not familiar with, but does the standard say anything about functions that never return? If it doesn't say that implementations may assume functions return then that sounds more like a compiler bug than taking advantage of UB.

shakna2• 6 years ago

Some compilers have a (non standard) keyword to indicate that a function will not return

The _NoReturn keyword was added in C11, standardising it. You can also get the now-standardised noreturn macro from stdnoreturn.h under C11, to maintain that compiler syntax.

moring• 6 years ago

Even if you assume the function returns, I don't quite understand why the if can be optimised away. The function may still have side effects if it returns, which have to happen before whatever assigning to NULL does. Those side effects may include sending data over the network, or happen in the operating system which is protected against whatever the program does, so even UB cannot affect it.

I guess UB is not assumed to include time travel...?

gpderetta7• 6 years ago

In C++ the function could throw, so at least there's that.

Also I believe that, because of posix, compilers will try to preserve side effects before potential UB. At least I couldn't get GCC to optimize out the function call.

jancsika2• 6 years ago

You get this bug because the compiler doesn't know that the function will not return and therefor assumes that the de-reference will happen even if the value is NULL.

That would be even worse reasoning.

If the compiler doesn't know that the function will not return then how could it possibly implement optimizations that remove this code?

Not returning from a function is well-defined behavior in C.

Dereferencing a pointer after a NULL check condition which doesn't return is well-defined behavior.

I understand that changing an "implicit else" to an explicit else will clue modern compilers in to the fact that the code shouldn't be removed. Regardless, removing the code in the author's example is dead wrong (quite aside of what one makes of Linus' quite persuasive argument about not even touching NULL checks which follow undefined behavior).

Edit: clarification

Edit 2: Ok, I need a sanity check.

Let's forget about functions which never return. Instead, let us change "write_error_message_and_exit();" to "return 0;" which will signal an error message to the caller. Also, assume there is a "return 1;" somewhere below the author's example.

Is there any compiler at any optimization level which would optimize out my revised conditional?

lmm9• 6 years ago

I thought Linus' rant was about NULL checks that came after a dereference had already happened. In that case it at least makes sense that the compiler would assume the NULL check would be superfluous.

IIRC it was an early return from the function rather than an exit function, something like.

    SOMETHING settings = *p;
    if(p == null)
      return;
    do_dangerous_stuff();

And the compiler produced something that unconditionally did the dangerous stuff.

aw162110719• 6 years ago

At least if I'm thinking of the same bug the code was this [0]:

    static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
    {
 struct tun_file *tfile = file->private_data;
 struct tun_struct *tun = __tun_get(tfile);
 struct sock *sk = tun->sk; // Dereference before null check!
 unsigned int mask = 0;

 if (!tun)
            return POLLERR;

I haven't been able to find Linus's rant, so I'm not totally convinced that I have the right thing.

[0] https://lwn.net/Articles/342330/

olliej• 6 years ago

There are many people arguing specific semantics of the authors arguments, but I believe the core problem is C and C++ both dramatically overusing “undefined” vs “unspecified”.

The difference is huge. Signed integer overflow is (per spec) undefined behaviour, so an obvious bounds check if UB and so can be removed. If it were unspecified the compiler would be required to be at least self-consistent. Eg it couldn’t do 2s complement in one place, but then treat arithmetic as not being 2s complement elsewhere (the overflow checks). E.g if the compiler emits code where MAX_INT+1 is MIN_INT, then the compiler can’t also pretend that that doesn’t happen.

Undefined should be reserved solely for things that cannot have a specified behavior (UaF, OoB memory, IO weirdness, etc).

quelsolaar10• 6 years ago

Some compilers even make it hard to test for overflow:

if(a + 1 < a) printf("Overflow error!"); else a++;

gets converted to:

a++;

because the compiler thinks that a + 1 can never be smaller then a since it doesn't have to consider signed overflow.

CJefferson• 6 years ago

While that looks stupid in isolation (and I agree it's annoyingly hard to check for overflow, although gcc and clang have special builtins nowadays to do it), it turns out there are important reasons for that optimisation.

In general, knowing that 'a+1' is '1 larger than a' allows for lots of optimisations, when writing to an array in order we can vectorise, do things in bigger chunks, all sorts of useful and important optimsiations. If every time those were used the compiler had to check for overflow, it would seriously effect performance.

brianpgordon• 6 years ago

The point of a compiler is not to try to show off that who ever implemented it knows more loop holes in the C standard, then the user, but to help the programmer write a program that does that the programmer wants.

The author makes it sound like the people working on optimizing compilers are deliberately seeking out these weird corner cases and selecting some random surprising behavior for them out of a hat, gleefully imagining how confusing it will be for end users. That's not how it works. Optimizers can be extraordinarily complex and need to maximize this ill-defined thing called "performance" in a highly multi-dimensional solution space. They ping-pong around inside this space constrained only by the specific requirements of the standard, and it's not surprising that some of the techniques used would produce some counter-intuitive results if the programmer is breaking the rules and relying on undefined behavior. It's kind of like if you trained a neural network to classify cat and dog pictures, and then you showed it a picture of a fire truck and expected it to give you a useful result.

The idea of a new version of the C standard that defines some of the most surprising undefined behavior is an interesting one though, and I'd be interested to see how much that really impacts the ability of the optimizer to do its work.

SAI_Peregrinus2• 6 years ago

I'd love it if the C standard just removed undefined behavior, replaced explicit instances ("the behavior is undefined" to "the behavior is implementation defined") and put in a blanket "Any behavior not specified by this standard is implementation defined". Keep the rest the same, just document the footguns. Implementation defined is exactly as powerful as undefined, it just makes the compiler writer describe what will happen.

dirtydroog• 6 years ago

It's C++, but UB allows the compiler to call a function the programmer never called

https://kristerw.blogspot.com/2017/09/why-undefined-behavior...

I'm not sure how any compiler writer can condone this behaviour.

aw162110719• 6 years ago

That particular behavior (and some other examples of pathological optimizer behavior) is probably an emergent property of how several optimizations combine as opposed to being a specifically targeted optimization. I can imagine something the optimizer doing something like:

1. Determine possible function call targets

2. Eliminate invalid targets from call set

3. If only one target remains, replace indirect call with direct call

That might be useful for something like devirtualization, but also could lead to the behavior exhibited in that blog post.

Indeed, the second blog post [0] shows that if there are two possible call targets, LLVM could generate an indirect call like what you may expect.

[0] https://kristerw.blogspot.com/2017/09/follow-up-on-why-undef...

AshamedCaptain• 6 years ago

That is not "never used" as the programmer argues, and I find it misleading that the title of the blogpost tries to sell it as "a never used function is getting called".

The NeverCalled function has external visibility so it could be eventually called by another CU, and it definitely references (or takes the address of) the supposedly unused "EraseAll" function. In order to be a legal program, some other CU has to call the NeverCalled at some point before main(), and the compiler is just assuming that.

scoutt2• 6 years ago

The article kind of lost me at the first example.

  struct{
    char x
    char y;
  }a;

  memset(&a, 0, sizeof(char) * 2);

What does the compiler know about memset? Why it should? It can be a function called heyLetsFillTheMemoryOverHere(&a, wrongSize); and still it's the programmer's fault.

In fact, it's a good thing that C doesn't care about memset, and doesn't get in between me and my code. And that's I love about C. It minds its own business and does what I tell it to do.

If I use my compiler to compile a program on my machine, the compiler knows that I’m compiling it to the x64 instruction set

You can be cross-compiling. I do it every day. The C language doesn't need to know about every little platform out there. If a compiler wants to go the extra mile and help about it, that's a compiler-thing and not a C-language-thing.

If there is a MCU that uses 3 bytes words, why should C be aware of it, 40 years before?

Don't blame the specifications, blame the compiler if you want.

Edit: removed a couple of personal opinions.

saagarjha15• 6 years ago

What does the compiler know about memset? Why it should? It can be a function called heyLetsFillTheMemoryOverHere(&a, wrongSize); and still it's the programmer's fault.

Because memset is defined by the ISO C standard to have specific semantics, which the compiler can leverage to make your code faster.

scoutt2• 6 years ago

void memset(void s, int c, size_t n);
The memset() function shall copy c (converted to an unsigned char) into each of the first n bytes of the object pointed to by s.

The standard just declares an interface, to be added to a C library (maybe?). It's not the job of the compiler to check for programmer's faults. The compiler doesn't care and it shouldn't.

MaxBarraclough3• 6 years ago

Certain standard-library functions actually do seem to get special treatment by the C standard.

memcpy and memmove get special treatment regarding effective type. I don't think there's any such special treatment for memset.

https://en.cppreference.com/w/c/language/object#Effective_ty...

Don't blame the specifications, blame the compiler if you want.

We're not talking about bugs in compilers (in which case you can blame the compiler), we're talking about standards-compliant compilers (in which case you blame the standard).

OliverJones• 6 years ago

rms was all over this "undefined" stuff. Early gcc versions actually tried to launch the games Rogue or Nethack upon detecting #pragma directives.

Old IBM "theory of operation" books used the word "unpredictable" for this kind of thing. "unpredictable" should frighten programmers. It's worse than "random," because random results have a chance of being caught during unit or system test.

lonelappde• 6 years ago

Why is a completely wrong blog post based on a misunderstanding of a 30 year old standard on the front page with 110 votes?

saagarjha15• 6 years ago

It's a perennial Hacker News topic.

throwaway20482• 6 years ago

because a lot of users here think that a compiler somehow detects undefined behavior, then emits wacky code just to spite them.

imtringued2• 6 years ago

I often see people still choose C even in 2020 for some projects. The usual arguments are that C has stuck with us for several decades and that it is now rock solid. Articles like this make me think otherwise. Why do modern compilers break old code and then claim that it was broken from the start?

saagarjha15• 6 years ago

Because the code was broken from the start, and previous compilers just didn't have the machinery to recognize that or do anything useful with it.

aw162110719• 6 years ago

Why do modern compilers break old code and then claim that it was broken from the start?

Because strictly speaking that old code is broken in that it violates the standard. Older compilers either didn't use the more aggressive interpretation of undefined behavior or didn't implement the optimizations that resulted in bad runtime behavior.

jcranmer• 6 years ago

The code was broken from the start.

Most of the undefined behavior in C arises from traps: if your code, naively translated, would have resulted in a trap, then it is undefined behavior in C.

Preserving when--and even if--a trap occurs is generally not desired, even by most end users. The instructions most likely to cause traps (memory operations and division operations) are expensive, and hoisting them outside of loops, or dead-code eliminating them is incredibly desirable.

When people complain about "broken" undefined behavior, what they usually mean is that they expected to get a very particular kind of trap and the compiler didn't give them that trap.

kps3• 6 years ago

Contrary to your other replies — old code, written in K&R C, was not broken from the start. ANSI C broke it with undefined behavior (by accident, I think) and ‘volatile’ (not by accident, just not fully thought through).

madmax962• 6 years ago

Compiler sanitizers and tools like Valgrind can go a long way to help developers understand why a program is broken.

einpoklum• 6 years ago

The author is encouraged to look into undefined behavior sanitizers:

https://duckduckgo.com/?q=ubsan+sanitizer&t=ffsb&ia=web

clang and gcc have those.

Other than that - @_kst is correct, it was the same in C89 and C99, and the intention is simply to let compilers make the optimization they like ignoring what happens for UB cases.

One Word Broke C

Related Stories