Getting Past C

http://blog.ntpsec.org/2017/01/03/getting-past-c.html

by ingve • 10 years ago

461 486 10 years ago

blog.ntpsec.org

falcolas7• 10 years ago

How much concern would non-standard architecture support matter for ntp? Given how many architectures Linux supports, I would think that C would still be the best choice, until these other languages gain support for those missing architectures.

Or perhaps it's a good opportunity for a language which offers transpilation with ANSI C as the target?

squiguy7• 10 years ago

Rust has impressive support for different architectures. I ran `rustc --print target-list` and it returned 63.

falcolas7• 10 years ago

That includes one line per architecture / platform (i.e. OS) combination. I only count about 14 architectures, though my rustc is old (GCC has 23 major, 24 minor, and 30 legacy as a point of comparison).

Last I checked, some of the popular IOT architectures (atmega, etc) were not included without some 3rd party plugins.

jerf• 10 years ago

In addition to squiguy7's point, I'd add that A: it may not be clear if this is your only contact with ntpsec.org [1], but this is actually a fork of the classic ntp, so they may be more willing to abandon some older architectures to produce a more secure product going forward than the core NTP project would and B: the older versions will still be around even so.

Also, it has been suggested by some experiences that older architectures often end up getting supported past when they really should be stopped out of sheer inertia, though I'm not having luck digging up the articles that prompt me to say this. Are there that many systems out there running ntp that can't run Go and/or Rust, and if there are, are there enough to be worth bending the project around? If the people running those things care, perhaps they should fork the project and maintain it themselves. Which is less harsh than it may sound, because they can still pull from upstream, and they probably just need to tread water rather than stay up with the latest & greatest.

(I should emphasize that the operative question is are there enough to be worth bending the project around, rather than whether there are any. Because there certainly are non-zero numbers of systems running ntpd that can't run Rust or Go. But who's going to pay to maintain them? Especially if classic ntp is still available?)

[1] https://www.ntpsec.org/

falcolas7• 10 years ago

Well, it's not just older architectures which are not currently supported, it's architectures used in IOT devices, mainframes, and other non-commodity hardware. Specific to LLVM based languages, if it's not a priority to Apple (or the other big LLVM players), it infrequently gets done.

Specific to IOT devices, I would personally love to see secure everyting. NTP, TLS, SSH, etc.

tedunangst4• 10 years ago

I'm never quite sure where MIPS and SPARC are in terms of llvm support. (And does llvm support automatically mean rust support?)

aaron-lebo• 10 years ago

Outside of the language war bubble it's really great to see a post like this. Practical concerns, reasonable advantages/disadvantages of each language, a real project dealing with real timelines. Thanks!

eduren• 10 years ago

I'm excited to see where this goes because it could go a long way towards providing concrete data for the large "work to replace old infrastucture C code with (Rust||Go||Modern C++)" discussion that has been taking place.

More data points will help to inform discussion, or at the very least add structure to the flame wars.

jstimpfle8• 10 years ago

Was going to say the same. In the past ESR has come across as a patently arrogant gun maniac, but the first part of this post is great for all the reasons you mentioned.

(What irritated me though was the switch to first-person narrative at the end).

lacampbell18• 10 years ago

One of the medium-term possibilities we’re seriously considering for NTPsec is moving the entire codebase out of C into a language with no buffer overruns

For a small one time fee of 1000 USD I can copy paste you 50-100 lines of C that provides an implementation of an array without buffer oveflows. Cheaper than switching to rust or you know, learning C properly.

noselasd• 10 years ago

Can you pass this buffer to e.g. read() and it will still guarantee no buffer overflow even if the programmer mis-calculate the size argument to read() ?

falcolas7• 10 years ago

You can with fixed sized buffers; the size passed to read() would be constant.

I'd argue that for programs such as NTP, avoiding dynamic allocations on heap is not such a terrible goal.

lacampbell18• 10 years ago

No.

It then follows that reading things safely is therefore a completely insurmountable problem in C. There is no possible way for me thin wrapper around read that works on bounds checked arrays, and not use "read" cavalierly in my code without thinking. Right?

civility6• 10 years ago

I'll bet he can give you an array_read() function that does what you ask. I know I could.

imtringued3• 10 years ago

Sure, if the array index is limited to 32 or 16 or 8 bits I can also give it to you at no performance cost for heap allocated arrays (i.e. no bound check happen at runtime).

There is some memory overhead per array with this method though.

jstewartmobile4• 10 years ago

A 62 KLOC secure NTP server seems like an ideal project for this kind of experiment. I imagine it would be self-contained enough to actually use Rust or Golang instead of just treating them like FFI scripters.

tptacek• 10 years ago

This is literally the only thing I can think of that "NTPsec" can do that would result in the project having any relevance. I understand why some very specific sites are chained to the ntpd codebase, but the vast, overwhelming majority of the ntpd deployed base not only isn't tied to ntpd, but also doesn't need 99% of what ntpd does. Trying to "secure" that codebase always seemed to me like a very silly windmill to tilt at.

pklausler• 10 years ago

Honestly, why not do it once and for all in a strongly typed pure functional language, validate it, and then tweak the GC parameters to get the performance? Use the safest and most powerful language that you can, if you can.

Ericson23144• 10 years ago

Yes, please use Haskell or Rust for this. Go is an awkward middle ground.

qznc2• 10 years ago

That would be a theorem prover like Isabelle or Coq. It is orders of magnitude more work.

kelnos2• 10 years ago

From my understanding by reading the OP, it's not really performance that's the problem, it's just that there are some (fortunately very small) time-critical sections in the NTP code that cannot tolerate a GC pause. If the language runtime supports disabling GC in critical sections (like Go does), then it's doable. If not, the language is likely automatically disqualified.

jstimpfle8• 10 years ago

Some people don't believe in tweaking GC's. I've heard horror stories from people who tried it on the JVM.

(And I have made terrible experiences with JVM's GC, but never tried tuning).

Some people don't believe that extremist functional languages are needed (depending on the domain).

Validation. Oh, well.

I think you propose Haskell, and it might disqualify for an NTP daemon, for example in terms of debuggability (stack traces?).

nathyong• 10 years ago

A more relevant point that would disqualify Haskell is the fact that its runtime (or rather, the GHC's runtime) is nowhere near predictable enough for the realtime guarantees that time synchronisation software would require.

A Haskell DSL that outputs some safer low-level code would be a more likely choice (for example http://hackage.haskell.org/package/atom), but Rust is both more popular and has more commercial support.

staticassertion11• 10 years ago

I think this is addressed in the article when they talk about the potential pitfall of Go's GC.

Peaker• 10 years ago

GC could possibly expose a program to timing side-channel attacks.

decebalus1• 10 years ago

strongly typed pure functional language tweak the GC parameters to get the performance

Care to give an example of such a language which allows for a guaranteed GC pause\resume configuration\declaration\etc in a critical section?

comex• 10 years ago

F# (i.e. the CLR) has this, although it's not pure:

https://msdn.microsoft.com/en-us/library/system.gc.trystartn...

dreta5• 10 years ago

Can anybody make a strong case to me as to why are buffer overflows considered an issue in C when it takes like 10 minutes to write and test an array implementation that prevents that from ever happening? I do agree that C has issues (though in my opinion neighter Rust nor Go address almost any of them) i just don't understand why are buffer overflows such a huge problem in C when the same thing is going to come up when trying to work with memory in Rust.

qznc2• 10 years ago

Libc does not use your safe array. You cannot pass your safe array to read and other syscalls.

tedunangst4• 10 years ago

As others have noted, one can also write a four line saferead() function.

dreta5• 10 years ago

No language you use can get around syscalls, so i have no clue as to why this is a case against C.

strictfp• 10 years ago

Because noone ever does that? Or at the very least people don't think of that as idiomatic c.

JKCalhoun• 10 years ago

Or perhaps just some helper functions in C that wrap array and pointer allocation/access to provide sanity checks. Seems like moving to a new language is rather extreme....

jdmichal10• 10 years ago

Why? Rust includes index checks by default, so you have zero chance of programmer error. And, to boot, it removes the checks when the compiler decides that the code is provably safe.

viraptor• 10 years ago

Sure, glib for example contains all the wrapped network / file / string handling you need. But then you need to make sure you use it correctly. And that no code goes around that API. And that you didn't make mistakes that invalidate the safety of your abstraction.

Moving to an abstraction would potentially solve some overflow issues and requires constant attention. Moving to a safe language likely solves all of them and you get checked by the compiler instead.

geofft3• 10 years ago

If you want to do this well, as other commenters pointed out, your entire C library interface is going to have to change to accept bounded arrays instead of pointers and lengths as separate arguments, or worse, just pointers and implicit lengths by looking for the NUL character. But then you've given up the ability to directly call into existing code; everything is at best using a little translation layer to check bounds before/after passing buffers to outside functions. So the advantage of staying in the language is minimal (note that you've given up on "C strings" entirely), and if you're going to have all of this, you might as well use a language with compile-time checks (types, etc.) that you're doing all of this right.

Rust and Go are not your only choices here. C++ with aggressive use of modern features and aggressive non-use of everything inherited from C is also a fine choice, but enforcing that discipline is hard, and the compiler won't help you a lot. Honestly, a dynamic language like Python or Ruby is also a fine choice in many cases, although NTP might be too latency-sensitive for that to work. (But it might not be! Premature optimization and all that.)

naasking3• 10 years ago

i just don't understand why are buffer overflows such a huge problem in C when the same thing is going to come up when trying to work with memory in Rust.

False. Buffer overflows in C can overwrite the program's memory, so it can be hijacked and supplanted with the attacker's code. This cannot happen in Rust (unless unsafe code has the vulnerability), or any memory safe language.

Sure you can implement a safe array/buffer abstraction and use it in your C programs that abort on invalid indexing. Now how many actually do this? Very few given the prevalence of C programs on vulnerability disclosure lists.

lossolo6• 10 years ago

I think he means to work with raw memory so using unsafe keyword and in this case he is right. And you can't implement certain things in Rust if you are on the quest for maximum efficiency without using unsafe.

jimmaswell• 10 years ago

How much does that still happen with DEP and such? Don't OSes not let you write to executable regions or execute from stack/heap by default now?

staticassertion11• 10 years ago

Because your 'safe' implementation will certainly have a performance cost, and won't be the default. This is why, despite C++ providing std::array, you'll still find buffer overflows in C++ code. C++'s std::array provides the safe 'at' function but you're opting into a performance penalty and it's not the more familiar [] syntax.

Rust arrays/ vectors are safe-by-default. To use the unchecked, unsafe version requires using the 'unsafe' keyword.

let v = vec![0, 1, 2]; unsafe { let x = v.get_unchecked(5); }

This means you can basically grep audit for vulnerabilities, and the above code should be very rare.

jdmichal10• 10 years ago

In addition, the Rust compiler can also remove the built-in indexing checks if it can prove the code is safe. So, say, iterator loops over an array won't have any index checking.

lacampbell18• 10 years ago

Would you care to provide a short rust implementation of reading an arbitrary length string from standard input?

I have rust installed. I would be curious to benchmark it and see if is indeed faster than the same thing in C, using a user-defined bounds checked array.

sidlls5• 10 years ago

How do you suppose runtime bounds checks are done in Rust? They certainly also incur a performance penalty in not-trivial cases.

Also, "safe by grep audit" means "safe according to a human." The argument of course is that it lowers the surface area of what a human must be trusted to verify. I'm still not convinced by that argument, because human error is a thing. And for actual systems programming, "very rare" may not be true.

sbov• 10 years ago

This is everyone's favorite excuse to trot out. But in reality the vast majority of projects I've seen never actually measure WHAT the performance cost would be and whether or not it's acceptable. So in the usual case the answer tends to be a mix of laziness and "that's how it's always been done."

mistaken• 10 years ago

To be fair the [] syntax is an overridable operator and you could just point it to use the "at" method. Not sure why they've implemented it as an usafe function.

EugeneOZ10• 10 years ago

No, Rust arrays are not safe by default: https://is.gd/iY5lPQ

jstewartmobile4• 10 years ago

When writing modern C in a disciplined way, it is not as bad as the Rustophiles will make it out to be, but still a problem.

Scenarios where I frequently end up fixing other people's memory errors:

1. No Error Handling: not checking an error condition on a function that allocates, then using the uninitialized pointer anyway

2. Sloppy Error Handling: jumping to abort from an error without freeing what has already been allocated

3. Faith in \0: still using the old string functions

I'm on the fence about the whole thing, so others may be able to field something more compelling.

pjmlp27• 10 years ago

Try fixing those errors in binary libraries or enterprise code.

moosingin3space6• 10 years ago

A properly-designed Rust API will not allow code without error handling to compile, so 1) should be much less relevant in Rust.

2) I can't remember if a panic in Rust calls destructors, which would clean up that memory. Can someone answer that please?

3) is only relevant in FFI scenarios in Rust, and everywhere else is irrelevant because Rust does not require the use of such footguns for basic string manipulation.

ssalazar• 10 years ago

I think it breaks down when that array has to interact with system libraries or the C stdlib in any way. A lot of C string functions have weird gotchas related to terminators and sizes, and any IO you're doing will involve raw buffers being passed into or out of a system IO function that doesn't understand custom array types.

jdmichal10• 10 years ago

Obviously, one can program C to do anything, and write all the provably safe abstractions wished. But, that's not really the point. The point is that doing such is not the default. It requires engagement and knowledge of the programmer, especially on distributed projects with loose communication, such as many open source projects. And it only takes one programmer mistake to bring the whole house of cards down.

Why allow programmers to make mistakes? That was fine in the 70's when resources for compiler execution were limited. I don't see any reason for it today.

I mean, just look at the underhanded C contestants and especially winners for ways in which your program can completely blow up for extremely subtle reasons.

panic• 10 years ago

And it only takes one programmer mistake to bring the whole house of cards down.

Isn't this also true of Rust, with its unsafe keyword? None of these languages are completely safe against programmer mistakes.

pjmlp27• 10 years ago

Even in the 70's there were better alternatives outside AT&T walls, e.g. at Burroughs.

userbinator2• 10 years ago

Why allow programmers to make mistakes?

For a philosophical counterpoint: Why allow anyone to do anything that might possibly be incorrect, harmful, or otherwise perceived by some to be negative?

I've looked at a lot of the talk surrounding "safe/secure languages", "safe/secure programming", etc., and yet every time I've heard people preach about the benefits, I feel like I just vehemently disagree. At a very deep and fundamental level, I feel like somehow we are sacrificing something more important in the pursuit of this "safety", this seemingly overpowering desire to make everything completely safe, mindlessly constricted, and stifling. It's not just software; the whole "war on terrorism" irks me in the same way. I imagine a "completely safe" world, the ones these "safe software" proponents appear to be striving for, would be rather dystopian.

A quote that immediately comes to mind is: "Freedom is not worth having if it does not include the freedom to make mistakes."

ArkyBeagle6• 10 years ago

C is the new "goto".

Y'all please, please note that dreta said "... an array implementation that prevents that from ever happening..."

pjmlp27• 10 years ago

C is the PHP and JavaScript of the 70's.

ArkyBeagle6• 10 years ago

Final word on the subject - what we have is people who are trying to make Open Software Reputation Points by finding a problem and fixing it, rather than waiting to find a real problem and fixing that.

When the figure of "ten minutes" was used - that's really what it should be as a mean or median figure, with some long-tail outliers for knotty cases.

While I am (somewhat) sympathetic, I don't miss what it is - it's make-work. There's not some body of material on language, O/S and software design available today that wasn't available 40 years ago. People new better back then, and didn't do better.

But the endless, circular, Utopian arguments just don't cut it. It ain't the crate, it's the pilot. I am sorry that oh so many are put upon to actually - gasp - test their code but that's what this is.

The economics of a Rust or Go or whatever make perfect sense - in the long run. Just recall what Keynes said about the long run.

dbaupp17• 10 years ago

The lack of generics means your array implementation is either going to either:

- be implemented with macros and token pasting, and result in a ton of mental overhead because you'll have a pile of types like array_foo for an array of `foo`s, and array_bar for an array of `bar`s, along with a pile of corresponding `foo * array_foo_get(array_foo, size_t)` and `bar * array_bar_get(array_bar, size_t)` functions.

- or, have a runtime cost and lose type safety by storing void* and casting when accessing.

The first case is even worse than it sounds: e.g. I don't know how you handle arrays of types with spaces in them (like `unsigned char`, or `struct bar`) with a macro. And, we haven't even thought about const correctness yet, which would probably require having const_array_foo, const_array_bar (etc.) types defined too.

(And, of course, these only solve one facet of the problems with C's pointers: there's no way to defend against use-after-free or dangling pointers.)

dreta5• 10 years ago

No, you just allocate enough space to store an extra int at the start for the length, and return a typed pointer to the actual data. Then you need an accessor that checks bounds, if you want safe access. Both of these problems are solved by simple macros.

nwmcsween• 10 years ago

You're missing main glaring issue with parametric polymorphism, bloat.

lacampbell18• 10 years ago

C macros-faking-generics really aren't that bad (your "unsigned char" case is really easy to solve - use another macro). It's a bit goofy having different types floating around like "array_of_int", "array_of_float", but you can create them in a single line when needed, and once you create them, they work, and efficiently.

They're not an ideal solution by any stretch, but it's not the nightmare scenario you envision wrt generic data structures in C.

MaulingMonkey4• 10 years ago

Can anybody make a strong case to me as to why are buffer overflows considered an issue in C when it takes like 10 minutes to write and test an array implementation that prevents that from ever happening?

The CVE database. Just because you 'can' write such an array implementation doesn't mean you will, doesn't mean your third party libs will, doesn't mean any of your legacy code uses it, and certainly doesn't mean you will properly test said array implementation correctly.

The number of mitigations added to C compilers and OSes dealing mostly with C and C++ code. ASLR, W^X, /GS, -fstack-protector-all, AddressSanitizer, ... - note the lack of similar tools, or demand for them, for, say, JavaScript - despite it enjoying a similar ubiquity.

I ask this in bad faith: I encourage you to share a single nontrivial codebase which actually creates the abstraction you've described and religiously adheres to using it throughout. As to why this is in bad faith: I'm definining "nontrivial" here to mean using 3rd party APIs - which will operate on C style arrays, not your project specific safe wrappers - and thus by definition won't be "religiously" sticking to said abstractions when using said APIs. By these definitions, the codebase I'm asking for doesn't exist - by definition. Even relaxing the "third party" rule, I haven't actually worked on a nontrivial C or C++ codebase without buffer overflow problems.

Now, e.g. Rust will have the same problems when interacting with C APIs - and nontrivial programs will end up doing so eventually. However, by virtue of the language itself embracing safe-by-default, you're less likely to run into the same problems when consuming Rust APIs.

You can also use third party static analysis tools to ensure you're using a "safe C subset" (such as MIRSA C), but "nobody" does that.

agumonkey2• 10 years ago

It's crazy that it's not solved above the language level, if people really want zero cost abstraction and architecture friendliness at least tooling should check buffer logic and flag the binary in case Warnings have been ignored.

phkahler• 10 years ago

I was thinking about mentioning MISRAble C. Outside the infotainment system, you should never have a buffer overrun on any micro in your car. Part of that comes from a ban on the use of malloc(). There is still a risk of array bounds problems, but those seem easier to avoid and more likely to be caught in testing.

Why no malloc? It was originally due to the small memory sizes for code and data. The less standard library the better, and dynamic allocation may lead to heap fragmentation and a subsequent crash when malloc fails.

wangchow2• 10 years ago

Honestly, we need a few AI coders to replace most of the developers in the world and then this won't be an issue. Bounds checking arrays and calloc instead of malloc isn't rocket science. It's a simple formula.

The problem isn't the language it's the developers.

scott_karana2• 10 years ago

Hate to tell you, but JavaScript implementations virtually all rely on C or C++ as well.

And it's not limited to the VM itself: check out npm "native extensions" like `json`. Not to mention glibc, or the OSes themselves.

By your definition, nothing is safe. And you're right ;)

Nacraile2• 10 years ago

I ask this in bad faith: I encourage you to share a single nontrivial codebase which actually creates the abstraction you've described and religiously adheres to using it throughout. As to why this is in bad faith: I'm definining "nontrivial" here to mean using 3rd party APIs - which will operate on C style arrays, not your project specific safe wrappers - and thus by definition won't be "religiously" sticking to said abstractions when using said APIs. By these definitions, the codebase I'm asking for doesn't exist - by definition. Even relaxing the "third party" rule, I haven't actually worked on a nontrivial C or C++ codebase without buffer overflow problems.

I work on a C codebase that does this, although in the slightly weaker sense that it does drop the abstractions at a few isolated interaction points with external APIs (think openssl, linux system calls, and not a whole lot else). Yes, there is quite a lot of NIH. With essentially-uniform use of checked data structures, and an extremely comprehensive suite of automated tests getting run under ASAN (originally Valgrind), memory safety errors almost never get so far as being committed to the main branch. This is a complex, >1M SLOC distributed system that has seen several years of production use at this point, and as far as I can recall we have not seen a single memory safety related issue in production (a few have managed to to get as far as certification testing). General resouce-leak class issues have struck a few times, but are also pretty rare.

Proprietary, naturally, so I can't actually show you (sorry), but it absolutely can be done in practice. It isn't even really all that difficult, it just needs to be done from the start, and then you just need a bit of discipline to keep it up.

pcwalton• 10 years ago

Because:

1. Buffer overflows aren't considered the most insidious issue in C nowadays. That award would probably go to use after free, which is not so easy to fix.

2. In C, it is easier and faster to do the wrong thing. Compare "char buf[256]; strcpy(buf, foo); ..." to "array_t buf = array_create(strlen(foo) + 1); strcpy(buf.ptr, foo); ... array_destroy(buf);"

3. Buffer overflows do not in fact come up routinely in Rust the way they do in C.

thesmallestcat4• 10 years ago

I'm no C expert, but those do two different things (fixed vs. dynamic length, stack vs. heap alloc). And the first one is safe with strncpy, right? Though I don't know why you'd ever have code like this unless you like undefined behavior and want to return the array. The second example seems fine and not particularly difficult or verbose.

dibanez• 10 years ago

Its useful to be able to remove safety checks for speed. I have a C++ code where all data is in array objects. Bounds checking is a compile time option, and it makes the overall code 2X slower. I can do testing with bounds checking on, but once it gets to a supercomputer that needs to be removed. Address sanitizing by compilers is an even more effective tool for this, especially for C. Bounds checking is critical for security, but if you're only concerned with correct execution then a segfault is not much different from an exception.

ChemicalWarfare• 10 years ago

technically you probably could limit yourself to using a "safe" subset of C - basically no pointer arithmetic, no strcopy() etc - but that would defeat the purpose of using C in a first place.

Sanddancer• 10 years ago

Compiler vendors have been resistant towards putting in such features. Bounds checking slows things down, and the performance race is very much a thing in C compiler implementations -- a compiler that can deliver a few percentage points better code can be a big win to teams working on compute heavy problems. C11 has Annex K which has a lot of safety features, like memory safe arrays. Unfortunately, none of the vendors have implemented it even as an option. Which is a shame because it would solve a lot of problems, with requiring minimal rewrites for a lot of code.

pjmlp27• 10 years ago

Annex K was required in C99 and made optional in C11, it tells everything on how C vendors see safety.

Also it is actually a joke, since it still separates the pointers and length in two separate variables, instead of using some kind of struct.

The only thing it does is have functions with better semantics on the terminating nulls.

josteink• 10 years ago

Can anybody make a strong case to me as to why are buffer overflows considered an issue in C when it takes like 10 minutes to write and test an array implementation that prevents that from ever happening?

That it's not a by-default and forced language-feature and that most developer aren't going to spend those 10 minutes when they need an array.

They'll just use the language-provided array-implementation instead. Which in C is very, very unsafe.

gsdean• 10 years ago

I believe your post is the strong argument you desire aka hubris

asdaksdhksajd2• 10 years ago

why does it take 62KLOC of C to distribute time? thats more code than the whole plan9 kernel.

asdaksdhksajd2• 10 years ago

doom was 52KLOC of C.

chungy• 10 years ago

Less than that even:

  all            37641 (100.00%) in 135 files
  c              37311 (99.12%) in 66 files
  asm              232 (0.62%) in 1 files
  makefile          98 (0.26%) in 2 files

signa11• 10 years ago

why does it take 62KLOC of C to distribute time?

as a starting point look here: http://www.ntp.org/ntpfaq/. you can go from there to the rfc's (5905 - ntpv4) to get a more indepth look.

dhsjxhx3• 10 years ago

Rather than looking for a new language, why not ban programs from writing to executable memory?

twic• 10 years ago

Because that isn't enough:

https://en.wikipedia.org/wiki/Return-oriented_programming

bvinc• 10 years ago

I'm not an expert in this field, but it's my understanding that this is often the case for C programs executing nowadays. The AMD64 architecture even has the NX bit. There are some other ways that this can be enabled too.

But surprisingly, this doesn't solve all code execution security problems from buffer overflows. Sometimes exploiters can find non-executable memory to change that allows them to change the program behavior to do what they want, such as changing the command string that gets passed to a normal execve call later in the program.

But even if you solve the security issues, buffer overflows are still a huge headache. A one byte buffer overflow can cause your program to crash an hour later, with almost no hope of figuring out why it happened. A developer can easily spend weeks tracking down a single buffer overflow crash.

dhsjxhx3• 10 years ago

A one byte buffer overflow can cause your program to crash an hour later, with almost no hope of figuring out why it happened.

I don't know about rust, but you can have an ArrayOutOfBoundsException in Java, and equivalent in Go and other safe languages

nullc• 10 years ago

Sometimes exploiters can find non-executable memory to change that allows them to change the program behavior to do what they want, such as changing the command string that gets passed to a normal execve call later in the program.

Changing function pointers or the return value on the stack are the normal things to do. In particular if you corrupt the stack you can simply return into system() with whatever arguments you want.

stusmall• 10 years ago

This wouldn't protect from bugs where you can maliciously trick a program from reading from other places in memory so it will leak passwords, keys etc. Think heartbleed

jjnoakes5• 10 years ago

That is done already in many cases.

Note that doing that doesn't fix all of the problems that out of bounds array writes can cause though.

w8rbt• 10 years ago

I would start by getting the code to compile with g++, then begin migrating the dangerous C constructs to safe C++ constructs. IMO, that would be a safe, reasonable thing to do.

pjmlp27• 10 years ago

How would you prevent other developers to write C style code?

blub4• 10 years ago

Project leadership, coding standards, automated analysis and code reviews.

But realistically speaking it might be easier to change tools for a significant number of teams, because the software development community is in general poor at leadership and process. A rewrite seems more approachable for the average (not necessarily average skills-wise) dev compared to a change in attitude, self-reflection and incremental improvement at a department or company level.

Then the rust compiler will pummel everyone into submission. This can be a succesful strategy. :)

zzzcpan• 10 years ago

After reading this post the idea of a C-to-C translator that injects bound checking, etc. comes to mind. Such translator could be used by OS distributions to provide safety in the least intrusive way and possibly completely automatically for many C codebases they have in their repositories. Translating into Go or Rust, on the other hand, cannot scale beyond some individual projects, that decide to undertake such efforts. Mainstream C compilers could implement safety features too, but realistically it cannot happen, as it's not something most people care about. So, C-to-C translator might be a best bet with the most impact.

the_duke• 10 years ago

It's a shame, really, but if you could write a decent C to safe Rust transpiler, than Rust probably would not exist.

Lifetime inference etc is a hard problem, and if it could be done right now, compilers would just have static checks for all that.

There is a C to UNSAFE Rust transpiler, though: https://github.com/jameysharp/corrode.

jawnv6• 10 years ago

Similar work has been done, over a decade ago at that. http://people.csail.mit.edu/rinard/paper/osdi04.ps

Realistically, the situation where you have source access with known errors you're somehow barred from fixing seems vanishingly rare.

nickpsecurity4• 10 years ago

CompSci keeps making them, even open-sourcing some, but little uptake or improvements from the FOSS crowd. Here's two of the top ones:

http://sva.cs.illinois.edu/

https://github.com/jtcriswell/safecode-llvm37

https://www.cs.rutgers.edu/~santosh.nagarakatte/softbound/

duneroadrunner2• 10 years ago

As we've discussed, one issue is the big performance cost of these solutions. Anyone interested in working on an automatic C to SaferCPlusPlus[1] translator? It should address the performance issue[2]. And should be much more straightforward than these C to Rust/Go translators.

[1] shameless plug: https://github.com/duneroadrunner/SaferCPlusPlus

[2] https://github.com/duneroadrunner/SaferCPlusPlus-BenchmarksG...

Franciscouzo• 10 years ago

You can just use a sanitizer such as UBSan [1], or ASan [2].

They are usually used in debug builds though, since they have a 2-3x runtime penalty.

[1] http://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html

[2] http://clang.llvm.org/docs/AddressSanitizer.html

elchief2• 10 years ago

what's wrong with OpenNTPD exactly?

NelsonMinar2• 10 years ago

There's a summary of some of its limitations here: https://en.wikipedia.org/wiki/OpenNTPD#Criticism

One of them (lack of leap second support) is fresh on our minds since we just had a leap second. Some thoughts from the OpenNTPD folks are at http://undeadly.org/cgi?action=article&sid=20150628132834

Chrony is another alternate NTP implementation, here's their comparison vs ntpd and openntpd. https://chrony.tuxfamily.org/comparison.html

elchief2• 10 years ago

It sounds like the leap-second thing isn't that big of an issue though...

giancarlostoro3• 10 years ago

I wish more mention of D would happen. It is compatible with C and C++ libraries and features GC without sacrificing the good things of C and C++. I always loved the idea of Rust and Go but they are nowhere near C or C++ where it matters to me. D fits the bill, otherwise I just use Python. I like being able to design software in my own way as opposed to being told how to do it.

thesmallestcat4• 10 years ago

I thought D still required a GC? Not that that's the end of the world, but with such strict requirements I'm surprised it would satisfy you.

p0nce• 10 years ago

I thought D still required a GC?

Not strictly required. You can certainly disable it at the cost of more work (and understanding). Or live with it and reduce its impact as needed.

scriptkiddy• 10 years ago

Last time I took a look at D I was awed by how terrible it's documentation is.

cakes• 10 years ago

I very much enjoy using D and want it to succeed but this statement along with finding "everyday uses" for things (e.g. using zeromq) are huge drawbacks to buying into it. I'll still advocate but it's not easy sometimes

moosingin3space6• 10 years ago

The truth is the Rust leadership did the hard work of building a community around the language, making high-quality tutorials and introductions to the language all before it stabilized for 1.0. Many developers claim to dislike "marketing", but Rust did its marketing/evangelism and D didn't.

From a purely technical standpoint, Rust's borrow checker is able to catch data races, while D has no such functionality (unless I'm not caught up), which is a huge advantage in today's world of multithreaded applications.

Manishearth29• 10 years ago

Even in single-threaded mode D isn't too safe. Safer than C, but unless you're using the GC it is still unsafe.

eridius5• 10 years ago

What do you mean by "they are nowhere near C or C++ where it matters to [you]"? Specifically with Rust (because I already know why Go isn't necessarily a good replacement for C or C++). I know very little about D, one of the few things I know is that it is compatible with C++, which is neat, but that's surely not a concern when talking about C.

giancarlostoro3• 10 years ago

As far as being able to write code in a paradigm I'm comfortable in as opposed to being forced into one when the purpose of a language is to replace C++ (which is what Rust and Go both were aimed at). As well as being forced where to put the opening bracket like in Go, which at first didn't bother me, but after a while it became a code smell of itself. There's plenty of talk about Go programmers copying and pasting code, which violates the DRY principle. I like having the option to use GC in D and still being able to compile my code. I even converted some C# code over to D that a project of mine heavily relied on and it compiles perfectly fine regardless of distro or OS I compile it from, granted it was very small / minimal code.

pjmlp27• 10 years ago

Me too, but although the community is great, lack of focus on how to go forward in memory management, runtime and deprecated packages, seems to keep hurting the language's appeal.

kqr3• 10 years ago

Same thing with Ada. If you're looking for a safer low-level language with a long track record of successful use by some major players, Ada is the natural first candidate.

But as someone else said, Rust has had way better marketing (and is newer, which probably catches some people as well.)

qz_• 10 years ago

Are there any large non-DoD projects written in Ada that are still used?

awinter-py5• 10 years ago

One such cleanup: we’ve made a strong start on banishing unions and type punning from the code. These are not going to translate into any language with the correctness properties we want.

Really? This sounds like idiomatic rust to me (heavy with enums).

Ericson23144• 10 years ago

C unions have no discriminant. But yeah it's a pity—I'd try to hack up discriminated unions then. Or convert to Rust unions and then enums and remove unsafety.

awinter-py5• 10 years ago

C union declarations should correspond pretty closely to rust enum declarations.

It's a surprisingly important feature. Every large codebase I've worked on has clunky workarounds for storing heterogeneous types in collections.

Haven't seen a silver bullet; dynamic languages are great at this until you have to scale (either in lines of code, number of types or dataset size) and then they become unmanageable. Static languages force you to type a lot and build in-memory ETL-style transformations but scale better. Using SQL is another solution, but that separates you from the language's type-checker and can be another source of error.

Type safety at the serialization boundary was the promise of CORBA and protobuf but we need better tools. I hope we see more focus on ser/des types in the next generation of industry languages.

rafinha2• 10 years ago

I didn't understand why rust and go are natural alternatives to C. Wouldn't C++ be a more natural option? (Despite the fact that both go and rust are developed by third party companies)

jimbokun• 10 years ago

"out of C into a language with no buffer overruns, and in general much stronger security and correctness guarantees."

Doesn't sound like C++.

shermanyo• 10 years ago

Really? It sounds like a good fit to me, maybe I'm missing something?

rafinha2• 10 years ago

I still don't get it. Can't it be done in c++ via appropriate data structures? I mean, it's not like Go is magic, at the end I suppose Go would be doing just that. As far as I know the main reason people opt for Go over C++ are the compiling times...

jstewartmobile4• 10 years ago

I did a lot of C++ years ago, so maybe things have changed since then, but I think Rust and Go addressed a lot of the design flubs of C++.

My experiences getting things to compile across gcc and visual c++, dealing with strings (especially Microsoft's WCHAR), reliable integer sizes (pre stdint.h), and debugging templates were not things I would wish on anyone.

Re-doing some of my side projects in Go and Rust was a lot more enjoyable. I could focus on what I was doing instead of trying to work around deficiencies in the language and its libraries.

AgentME• 10 years ago

None of the safety issues brought up about C there are solved by C++. C++ is (nearly) a superset of C, so it inherits all of those issues.

silent_hunter2• 10 years ago

If C++ is "(nearly) a superset of C" why is the C++ standard twice the size of C standard? Of course if we take that 0.505.. ≈ 1, then it is indeed a (nearly) superset.

blub4• 10 years ago

C++ would be a great alternative, assuming the developer(s?) working on the project didn't hate it, as is the tradition: http://esr.ibiblio.org/?p=532

Oh well, at least they're moving from C, which will be a big win either way.

ericfrederich• 10 years ago

I like go, I'd love to write libraries in it but as far as I can tell you can't really create a C compatible shared library from it. That it still the common denominator if you want to call into it from other languages. I'd love to write Python programs with performance critical stuff in Go

LukeShu• 10 years ago

    as far as I can tell you can't really create a C
    compatible shared library from [Go].

Sure you can! (as of Go 1.5, iirc)

Change your build command to use the c-shared build mode:

    go build -buildmode=c-shared

Then, to change the linkage of the functions you want to export, use the magical comment "//export" directive:

    //export MyFunction
    func MyFunction(arg1, arg2 int, arg3 string) int64 {
        ...

steveklabnik28• 10 years ago

This still doesn't get over the inherent overhead in calling between the two languages though, right?

(And you're bringing that whole runtime with you. Which might be okay! As always, it depends.)

nurettin4• 10 years ago

Why does 90% of this huge comments section revolve around Rust? Haven't we had enough of the same already? Yes, we know it provides memory safety guarantees. Yes we know you hate everything written in C with great passion. This has been made apparent and banged on our heads for the past several dog-years. Did anyone bother to look at the sheer amount of refactoring the author & team did? Did anyone realize how difficult this is and what they might learn from this tarrasquesque ordeal they went through? Oh no, apparently it is C and logic mandates that Rust supercedes it so why.the.hell.bother.

eridius5• 10 years ago

The huge discussion about Rust is because that's literally what this post is about. It's titled "Getting Past C" and most of the post is about considering Go or Rust as the future language for the project. And, not surprisingly, a lot of people think Rust is a great choice for this (and I agree).

nurettin4• 10 years ago

The title is Rust-bait so let's all chant rust! Rust! RUST! Oh but there are only three tentative mentions of Rust in it and this entire comment section is corroded. Just compare the two pages in a browser with ctrl-f rust

flukus• 10 years ago

Because if (a big if) we ever move beyond C then rust is looking to be the biggest contender.

pjmlp27• 10 years ago

Because before C got widespread beyond UNIX, we had lots of options that are now gone, so the millennials only know Rust as having those features.

Of course we also can talk about D, Go, Swift, .NET Native, Haskell, OCaml as possible modern languages to use instead of C.

Hint, many uses of C actually only care about AOT compilation to native code, and never have spent even 1 second measuring performance.

anthk• 10 years ago

Under Linux, some SECCOMP initialization and capability dances having to do with dropping root and closing off privilege-escalation attacks as soon as possible after startup.

Thanks, Eric. Now, FFS, learn how the Cathedral model WORKS, get OpenBSD, learn pledge(2) and stop spreading old FUD against C.

e3b0c• 10 years ago

Rust has some very desirable properties to me. Writing Rust programs from scratch is not as scary as I've heard of from the internet either. The documentation is excellent, the compiler diagnostic messages are very helpful and the notorious borrow checker didn't stand in my way that much. And I love Cargo and Cargo.io. I have some projects where Rust is the saner choice than Go or other GC based languages.

That said, there are actually drawbacks of Rust compared with Go, IMHO. When facing a moderately large project written by others, the ergonomics for diving into the project is not as smooth as Go. There is no good full-source code indexer like cscope/GNU Global/Guru for symbol navigation across multiple dependent projects. Full text searching with grep/ack does not fill the gap well either since many symbols, with their different scopes/paths, are allowed to have the same identifier without explicitly specifying the full path. That makes troubleshooting/tracing a large, unfamiliar codebase quite daunting compared with Go.

steveklabnik28• 10 years ago

RLS will hopefully full that gap.

rattray• 10 years ago

RLS?

wocram• 10 years ago

I haven't had too many issues with Intellij-rust or racer failing to jump between symbols.

There are also many other tools that provide indexing, eg. ide [plugins], kythe, and the rust language server.

xorxornop• 10 years ago

Hmm, I've had a very nice experience using Rusty Code in VS Code. Some useful refactoring functionality is missing for sure, but a lot of that will become possible quite shortly from RLS (Rust Language Server, a la how TypeScript works in VSC), and if your preferred editor has support for the language server spec (it's a open source common spec, not specific to Rust), it will support it at parity, too.

Manishearth29• 10 years ago

YouCompleteMe on Rust has pretty good JumpToDefinition support for Rust.

You can also use https://github.com/nrc/rust-dxr to index rust code via DXR.

IIRC ctags also works with Rust.

RLS should cover this pretty well too once it happens.

leshow2• 10 years ago

crates.io

Reply

acjohnson55• 10 years ago

It's not yet ready for primetime, but Scala Native (http://scala-native.readthedocs.io/en/latest/) might just make a splash in the systems space. I don't think it has anything like ownership yet, but I wouldn't be surprised if it eventually develops that capability. I think you can get it to run without GC, too, but using C Stdlib memory management. Although, that largely defeats the memory-safety.

Just throwing it out there as something to keep an eye on!

Reply

arunmu3• 10 years ago

I do not contest on the opinion that Rust is a good language, but it slightly hurts me when people club C and C++ together. One can easily write correct by construction code using modern C++. Use of meta-programs allows you to create typesafe constructs. It provides you with zero cost abstractions to specify ownership of resources and ..... <I can go on> One has to just strive to not use the C baggage that comes with it.

Reply

Manishearth29• 10 years ago

C++ is much better, but still not safe. Null still exists. Moves are runtime moves, so you can still attempt to access the value at compile time. Iterator invalidation is still an issue.

Of course, C++ might be "safe enough" for your use cases.

Reply

arunmu3• 10 years ago

Moves are runtime moves, so you can still attempt to access the value at compile time.

What do you mean by that ? Are you saying the standard decided to make it non-destructive moves ? Then yes, it is a possible scenario for error but also a performance advantage in some cases.

Reply

pjmlp27• 10 years ago

One has to just strive to not use the C baggage that comes with it.

This is why they get clubed together, regardless how much I like C++, I am yet to see the use of C baggage being successfully forbidden in enterprise teams, let alone if there are third party dependencies (which is always the case).

So far I have only seen modern, safe C++ being used successfully on a big project I was part of at CERN, where everyone on the team actually cared to write proper C++.

Reply

blub4• 10 years ago

Do those enterprise teams use rust / go or have they already moved on to Java and .NET many years ago? :)

The enterprise is not the target here.

Reply

z3t4• 10 years ago

Why do they have to use a low level language ? Why not use NodeJS ? One argument would be time management, but even with C you can not be sure that no interrupts happens in between ...

Reply

jaco8• 10 years ago

Looking at the current new and coming languages I would take a hard look at NIM. It may be is not there yet but it looks highly appealing, is as fast as the often mentioned Rust and compiles significantly faster. http://nim-lang.org/

Reply

nimmer• 10 years ago

Nim compile times are faster and often the execution time is faster, too. Also it's very expressive, almost like Python.

Reply

amadvance• 10 years ago

Rust is surely fine, and an improvement over C, but its main advantage is that all the rust code is written now, when everyone takes more care about security.

It doesn't have to deal with 40 years of bad legacy code written by sloppy developers.

You can obtain similar quality in a C modern code-base, using tools like static and dynamic analyzers. In fact, today the hardest issues came from multi-threading. I won't even dare to write multi-threading apps without helgrind/TSAN.

And Rust doesn't help in this regard. From: https://doc.rust-lang.org/nomicon/races.html 'So it's perfectly "fine" for a Safe Rust program to get deadlocked or do something incredibly stupid with incorrect synchronization.'

Reply

blub4• 10 years ago

Some people take more care about writing safe & secure code today. Most only care just enough to switch to Java or Python in order not to care any more.

The big question for me is what will happen when those same developers writing insecure C and C++ code start writing rust? Will they find ways to subvert the safety checks? Will they even want to subject themselves to them in the first place?

Reply

_yosefk• 10 years ago

I find it rather daring to use raw POSIX threads or similar, hoping that automated debugging tools operating at this abstraction level, such as Helgrind or TSAN, will pinpoint your bugs. The problem is that the state space of a thread-based program synchronizing with mutexes is too large to explore at a reasonable time. More about this: http://yosefk.com/blog/checkedthreads-bug-free-shared-memory... and http://yosefk.com/blog/making-data-races-manifest-themselves...

Which higher-level abstraction to use is a question and depends on the problem at hand; for throughput computing in C, Cilk is perhaps the best today (I guess you could call my own project checkedthreads a bit of a Cilk knock-off.) AFAIK, Rust developers are more interested in higher-level abstractions than Go developers (Go's built-in goroutines being on par with threads in terms of ability of automated debugging tools to flag bugs; perhaps there's work in Go on higher-level abstractions but the vibe I felt, perhaps mistakenly, is that these are non-problems, which I disagree with: http://yosefk.com/blog/parallelism-and-concurrency-need-diff...)

Reply

atilaneves• 10 years ago

Similar quality in a modern C codebase? No. Much better than in the past, yes, but not similar. Even if the quality _were_ similar, it'd be disproportionally more expensive to write.

I just left a job in which C code was being written from scratch, in 2016. The code was awful. Coverity didn't prevent it from being awful.

Reply

nsuser2• 10 years ago

And Rust doesn't help in this regard.

From: https://doc.rust-lang.org/nomicon/races.html

Data races are mostly prevented through rust's ownership system: it's impossible to alias a mutable reference, so it's impossible to perform a data race. Interior mutability makes this more complicated, which is largely why we have the Send and Sync traits (see https://doc.rust-lang.org/nomicon/races.html).

Reply

tannhaeuser• 10 years ago

My experience is that using C for main(argc,argv)-style programs is rarely a problem. Trouble comes when using long running single-address space containers for service-like abstractions with pthreads etc.; in that kind of environment, malloc() and co. don't cut it because even if you get memory allocation right, unless using pooled memory allocators, memory fragmentation is becoming a serious (ie. unsurmountable) problem.

It's been said over and over since at least the Java times that creating OS processes for individual service invocations is bad for performance, but I've never seen proof for this statement in the form of a benchmark.

Even the OpenBSD developers (who know a thing or two wrt. security of memory allocation schemes) diss process-per-service-invocation architectures in their httpd implementation (eg. calling their CGI bridge "slowcgi" and favouring fcgi over it).

Isn't that inconsequential? I mean if there's a performance problem with CGI-like process-per-service invocations, why not target these problems at the OS level (or via pooling of network connections or whatever the bottleneck is)?

Reply

eggy• 10 years ago

I didn't see Ada or ATS mentioned upon a quick read here. I don't program in either beyond exploratory play, but they seem to fit the need here no?

Reply

kcudrevelc• 10 years ago

Under Linux, some SECCOMP initialization and capability dances having to do with dropping root and closing off privilege-escalation attacks as soon as possible after startup.

I was under the impression that these specific things were actually quite hard to do in Go. I believe that both setuid/setgid and seccomp_load change the current OS thread (only), and since Go multiplexes across multiple threads and gives programmers very little control over which ones are used for what goroutines, I'm not sure how you would, for example, apply a seccomp context across all threads in a Go program. setuid/setgid are currently unsupported for this reason, with the best method being "start a subprocess and pass it file descriptors" (https://github.com/golang/go/issues/1435).

I'd be interested to hear if others have found ways to actually do this reliably for all OS threads underlying a running Go process.

Reply

plietar• 10 years ago

I did this once, by writing a small C program which sets up the seccomp context before exec'ing the Go binary. Unfortunately Go's runtime makes a huge number of system calls in the background, and the whitelist kept growing.

Switched to Rust and there was only had one hidden system call left, getrandom used to initialize the hashmap

Reply

Getting Past C

Related Stories