Linux eliminates the strncpy API after six years of work, 360 patches

https://www.phoronix.com/news/Linux-7.2-Drops-strncpy

by simonpure • 1 month ago

300 323 1 month ago

phoronix.com

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

mrlonglong• 1 month ago

the zero terminated string is I think is computing's biggest mistake. Pascal style strings were much safer.

jackbucks• 1 month ago

It was definitely an interesting way to allocate pointers. I did once have a very large project where devs didnt understand this and resolved hundreds or more off by one and memory overwrites in C due to this feature.

But at the same time, I think blaming the software was kind of a cop out. Devs were in a hurry and simply didnt respect the rules. Given todays software engineer at large. Nerfing programming languages so they cant destroy things might not be a bad idea. But AI will nerf everything.

fragmede4• 1 month ago

why is AI gonna nerf everything? sure it could be used as the easy button, but I just spent two hours this morning learning about the neuroscience of how memory works in the brain that I didn't mean to and now I want to run studies on how memory works.

Why do you assume that AI is gonna nerf everything?

dietr1ch2• 1 month ago

I think it was NULL itself. It was a long way until we realised we don't want invalid values and could use the type system to help us use special values safely.

jkercher• 1 month ago

Meh, I think NULL is fine in C. It's an extra, valid state to represent pointers at no cost. Unlike the more hand holdy languages, it's quite rare for a pointer in C to have the ability to be NULL since, more often than not, it's pointing at something known. It's actually quite rare to see NULL checks unless it's API code or something like that. I can see this being more of a problem in a managed language where anything can be NULL at any time.

bellowsgulch• 1 month ago

Compared to scripting languages with actual tagged types, C doesn't really have a type system, and that's readily apparent to anyone who has written C in the last 43 years and debugged a program written in it.

C pretends types exist with you, but once bytes hit the road, it's all real-life and segmentation faults.

jkrejcha3• 1 month ago

The problem here is that null kinda is consequential of intentional design of the type system itself. In this way, I do think that null was discovered, rather than invented. Remember, C is a kinda "portable assembler" so the constructs in it are based relatively closely to how low level data structures are mapped out in memory.

This is, and continues to be, an incredibly useful feature that makes C and C structs immensely useful concepts. Part of that does need an invalid value[1]. NULL is convenient for this and although there are some very weird JavaScript-trinity-meme-style consequences for this[2], it's such a useful concept that basically all languages that have the ability to construct pointers have a null pointer[3].

The alternative world looks like everyone inventing their own invalid values. Invalid, non-null, pointers are typically MUCH worse than null pointers for debuggability and security. If you unintentionally read/write/execute memory at 0x0 (by far the most common value for NULL), most operating systems will trap this, whereas may not necessarily if 0x12345678 is your invalid value.

[1]: Stuff like IA64 had NaT bits which were effectively an extra bit for what I assume to be this sorta thing. The problem with this is that it costs an extra bit. I don't really know much about IA64, but presumably [NaT 1] + [don't care] would be your null pointers here. I think?

[2]: Really what the standard, in my opinion, should have done is probably not make use of the null pointer UB for many different functions. A lot of compilers took the UB surrounding that to make incredibly dubious "optimizations" that broke stuff with zero actual performance benefit whatsoever

[3]: Yes, even Rust. Although some (again in my opinion) unfortunate design decisions made it so that C-Rust FFI isn't zero cost because of how it treats spans/slices

atherton940275• 1 month ago

Genuinely curious, how would you handle cases where a value is unset without NULL? This is a legitimate case that happens a lot in eg data modeling

themafia• 1 month ago

Pascal style strings were much safer.

The limitations were brutal. Initially you could only have 255 bytes in a string. The length of a string and the size of the allocation are now separate and you may need to think about that unused memory in your design. The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.

If you want to create an array of strings you either need to specify the length of all strings and accept the memory overhead or have an array of pointers to strings. If you use an array of pointers you may end up choosing to use the 'nil' value as a sentinel that means "end of list." So we're right back where we started.

Because someone decided to downvote this HN has limited the speed at which I can reply. This site is tragic and I'm fully done with it now. You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately. Screw this.

No other data structure works like this.

The linked list.

You can't mess this up in an array

C happily decomposes arrays into pointers. You can erase your length information from the type. This was an intentional decision.

Strings are the only data structure that assume there will be a NULL at end.

Which is why almost every string API has a version that allows you to specify the maximum length. The fact that you can use a NUL doesn't mean you have to. Which is why the concept of "sentinel values" is broadly used in many types of applications you haven't considered here.

AlienRobot3• 1 month ago

The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.

That isn't really a problem.

The problem with null-terminated strings is specifically what happens when you reach the end of the allocated array and there ISN'T a NULL character.

Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character, he can exploit pretty much any standard string manipulation function being used elsewhere in the program to manipulate whatever memory comes AFTER the string data structure.

No other data structure works like this. You can't mess this up in an array, because no function that manipulates arrays is just going to keep going until there is a null. That would be stupid because it would require users of the function to add a NULL to the end of their arrays before passing it to the function, so instead we just pass the size of the array to everything. Strings are the only data structure that assume there will be a NULL at end.

By the way, I read once that if you use UTF-32 every code point will be 4 bytes, constantly, but even then a single code point isn't necessarily a single character. Text is just complicated.

BigTTYGothGF• 1 month ago

Your string size is in bytes and you need to track characters separately

No worse than C strings then.

dare9445• 1 month ago

You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately.

Indeed. And the ignorance of computing history in this discussion is particularly disturbing.

The context of this particular thread is "zero terminated string is ... computing's biggest mistake". This completely ignores the situation on the ground when C was developed. At the time, people were striving for a system programming language that sat above the level of assembly but was compact enough to run within the limited resources of the then emerging mini-computer systems. The PDP-11 on which C was developed was certainly not the first mini-computer, but it was among the earliest to have a regular enough instruction set and addressing model to make a general purpose, high-level system's language possible. These systems were extremely limited in memory; the PDP-11's instruction set is limited to directly addressing at most 64KiB (code and data) and many systems of the era were hardware limited to less than that. (Indeed, I regularly run an early version of Unix, including an early C compiler, on my PDP-11/05 which is maxed out at 56KiB [of actual core]). There was no way that even a brilliant engineer like Dennis Richie was going to be able to shoe-horn in "optional" types, or the mechanics of length-value strings into a compiler that has to run in such limited space, and produce code (e.g. the Unix kernel) that has to run in even less. The fact that strings and arrays are thin abstractions on top of pointers is both a brilliant compromise in design as well as a nod to then-prevalent assembly practice. It was the exactly kind of pragmatic decision that was needed to move computing along at the time. Of course the designs from this era are antiquated now. But they were not mistakes.

pjc503• 1 month ago

You can erase your length information from the type. This was an intentional decision.

Well yes, but given the number of security issues the argument is that it was in retrospect the wrong decision.

pjmlp9• 1 month ago

All those limitations were sorted out in 1978 with Modula-2 and open arrays, aka spans.

What about the UNIX and C folks propaganda of C being the first systems language, or always focusing on the original Pascal used for teaching and not everything else that followed up with Mesa, Modula-2, Ada, Object Pascal and friends, none of them with said limitations.

msla• 1 month ago

In addition to having to pick a size for the length counter and then, later, having to differentiate between lengths in bytes, codepoints, and glyphs, you can't subdivide a Pascal string using pointer arithmetic. To pass just the end of a string into a function, you have to either copy the tail of one Pascal-style string to another with a smaller size value, or your string has to be a struct with an integer and a pointer to the actual data instead of just an integer stuck on the beginning of the string. The first is a lot of copying in some cases, the second raises the specter of structs with invalid pointers. That's not to mention the potential problems that would cause with caches.

estebank• 1 month ago

The third option is to have a variable width length: the top most bit signals whether the next byte corresponds to the length or to the start of the string.

cornholio2• 1 month ago

You can have a universal variable length field, for example 2 bytes for strings < 32768, then four bytes, 8 bytes etc. On the critical short string path, it costs just a single bit test. The glyph vs byte issues need to be dealt with in both formats.

The subdivision issue is a good perspective, but i would argue the performance impact of cloning substrings is dwarfed by the redundant full string reads to find length.

pjc503• 1 month ago

.. which is why you need a second type, the one dotnet calls "Span". A substring.

bsder• 1 month ago

Zero terminated string is a special case of sentinel value termination.

And sentinel value terminations make a lot of sense when you have punch cards and fixed length records that you need to carve into pieces.

Nobody expected any decisions they were making in the 1960s and 1970s to have any bearing on computing a half-century later. They all expected to have their mistakes long papered over by smarter people at some point.

But we ALL make the mistake of underestimating inertia.

fragmede4• 1 month ago

compared to Von Newman versus Harvard architecture for LLMs? I think that's a far bigger mistake.

pjc503• 1 month ago

Neumann, and .. what? In what way?

layer85• 1 month ago

Almost as bad as newline-terminated lines. ;)

sourcegrift2• 1 month ago

What's bas about them and what are the alternatives, genuinely curious since never seen them spoken of

smackeyacky3• 1 month ago

Zero terminated strings were the basis for an awful lot of useful software. Calling them the biggest mistake in computing is a bit OTT.

I haven’t programmed anything Pascal related for 30+ years but I dimly remember thinking at the time that I wished the string system wasn’t so hard to use.

asdfasgasdgasdg• 1 month ago

That useful software would not have been less useful if the strings in it were represented as size + buf.

ComputerGuru2• 1 month ago

That argument isn’t valid. The argument would be “this string design enabled a whole lot of useful software” but that’s a different matter. (And it could very well be the case.)

zzrrt• 1 month ago

Lead was the basis for an awful lot of useful gasoline. Doesn't mean it was the only solution or the best one.

JdeBP3• 1 month ago

A more accurate re-phrased version of the original is that they are the biggest mistake in the C language.

* https://news.ycombinator.com/item?id=48614913

* https://news.ycombinator.com/item?id=24454369

* https://news.ycombinator.com/item?id=1014533

saagarjha• 1 month ago

zero terminated strings may cause a lot of bugs, but it also ends up in a lot of useful software, so, it;s impossible to say if its bad or not,

dmazzoni• 1 month ago

255 characters ought to be enough for everybody, right?

RetroTechie• 1 month ago

You mean bytes.. we have multi-byte characters now.

BobbyTables2• 1 month ago

Partly agree but there would have been squabbling on the data type of the size, unless it was variable length. The latter would have had other issues too.

For a while, 16bit would probably have seemed too extravagant. Now 32bit would probably seem too small.

For a “strongly typed” language, C is pretty damn loose where would have mattered.

poly2it• 1 month ago

No, there would not have been and this is most likely not the reason. size_t exists for precisely this use case. It has existed since C89.

DarkUranium3• 1 month ago

I like the D approach where arrays are just `struct { size_t length; T* ptr; }` internally --- and strings are just arrays of `immutable(char)`.

It has a big advantage over the Pascal approach in that you can do zero-copy slicing, since the length is separate from the actual data.

And `size_t` makes perfect sense for the length here. If your strings are longer than the address space (which `size_t` technically isn't, but is practically very strongly correlated to it), then you're going to have a problem regardless of the number of bits for the length anyway.

zeroonetwothree• 1 month ago

C is not really strongly typed.

darkr• 1 month ago

C is a weakly typed language. It’s statically typed, which is a different thing

bigstrat20035• 1 month ago

I don't think 32 bits for the size would be too small. That would max out at a 4GB string, which is large enough that even today it's a big red flag saying "what are you doing bro, reconsider your approach". I can conceive of a string larger than 4GB, but I can't conceive of a situation where it's reasonable to use one.

Conscat2• 1 month ago

Clang and GCC both let you use Pascal strings in C if you would like (with `\p`). But Pascal strings aren't that useful today because the maximum length is too short.

jxbdbd• 1 month ago

Why would a pascal string be any shorter than a C string?

A C string is one pointer reaching all of memory, a Pascal string is two pointers reaching all of memory

lelanthran15• 1 month ago

the zero terminated string is I think is computing's biggest mistake.

No. They had trade-offs to make, and sentinel-based sequences are a needed thing, even outside of strings.

The mistake was that ISAs never looked at what HLL needed, then add the necessary instructions (I posted more about this below).

Even NULL is not a big mistake, when looked at in context of the time in which it was developed.

layer85• 1 month ago

There is a middle ground that Visual Basic (and then COM) took, with the BSTR type: It’s still a pointer to a zero-terminated char array, but there is a length field immediately preceding the first pointed-to byte. This is still compatible with a C string (assuming no embedded null characters), but BSTR-typed functions can take advantage of the length value.

tremon• 1 month ago

This is still compatible with a C string

Strictly speaking, it's not alignment compatible from CString to BSTR unless you declare all strings to be at most 255 characters or the cpu architecture doesn't require aligned access for multi-byte words (like x86). The BSTR alignment must match the alignment of the length word, meaning you can't convert a randomly-aligned C string to BSTR by simply attaching a prefix in-place.

Also, having the length embedded in the value rather than in the pointer makes it impossible to create BSTR (sub)slices without performing a memcpy. Fat pointers do not have this restriction.

jiggawatts• 1 month ago

These are great for "data smuggling" attacks where one layer of code assumes the length is 'x' and another layer assumes it is 'y'.

It makes hybrids like this very dangerous for anything even remotely security-adjacent, such as roles, tokens, etc.

This kind of thing caused the CVE-2009-2408 and CVE-2009-2510 "Null Truncation in X.509 Common Name Vulnerability."

badsectoracula3• 1 month ago

FWIW pretty much all Pascals since the 90s use a similar approach.

In Free Pascal for instance, strings are pointers to the first character with a header in a "negative address" containing information about the length, reference count (strings are reference counted and use copy-on-write to avoid passing around copies all the time) and codepage (FP can convert strings between different encodings "transparently").

pjmlp9• 1 month ago

Nowadays replaced with HSTRING.

sourcegrift2• 1 month ago

Rust has "pascal style strings" (quotes because the concept is slightly different) so it's not a done deal

larodi3• 1 month ago

Wonder when is someone going to brave and fork the linux kernel and try to ffwd it with automatic programming.

fragmede4• 1 month ago

why would you start there instead of creating something from scratch ?if you can port drivers just as easily meaning you don't especially give a shit about hardware you're running on in the first place, why even deal with linux? The battle tested LRU cache system?

literalAardvark• 1 month ago

It's much easier to use something with all the edge cases already handled as a starting point.

convolvatron• 1 month ago

I've seen several workalike kernels in various stages of completion. at least one of them was able to run some pretty substantial applications (Postgres, nginx, that kind of thing), and that is still I guess around 250kloc. but it only really has drivers to support hypervisor devices.

unfortunately as time goes by, the linux api surface gets larger and more convoluted. so there's going to be some coverage you're just never going to get.

but in the abstract, definitely. linux is so bloated at this point that its not clear that it can ever be 'made safe'.

larodi3• 1 month ago

Well in reality if you want a custom OS perhaps scavenging parts is a thing to do indeed. I just speculated whether Linux can be further improved by automatic programming and still keep the handmade parts.

bigstrat20035• 1 month ago

Going fast is only good if you're going the right direction. And with LLMs, more often than not you're going off a cliff.

PlunderBunny• 1 month ago

I worked on a Win32 app that used space-padded strings, i.e. the destination string was padded with spaces, but there was still a null on the last byte. You had to use special versions of the string functions for length, copy etc.

I’m not sure why this was - the source base was so old it might have had its origins in Pascal struct behaviour.

egorfine• 1 month ago

I think this behavior has its roots in COBOL, not pascal.

kps3• 1 month ago

Which has its roots in punch cards, where pre-computer hardware operated on fixed-sized fields and an unpunched column is equivalent to a space.

bebe83939• 1 month ago

Perhaps prevent realocation when string size changes? Or aligning cpu cache lines?

jkfkfkj• 1 month ago

It can perhaps be due to the string originating from a sql database ”char” field, I.e. not ”varchar”. Char fields in databases are space padded.

naturalmovement2• 1 month ago

A reminder that we've had strlcpy[1] for ~ 30 years but it was never accepted into the Linux world because of typical petty open source bullshit. This is why we can't have nice things.

[1] https://man.openbsd.org/strlcpy

BoingBoomTschak2• 1 month ago

Actually, glibc 2.38 has it.

naturalmovement2• 1 month ago

Wow it only took them 26 years to import a 30 line C function, a third of which is comments?

I should have sent them a nice fruit basket to commemorate the occasion.

ericbarrett• 1 month ago

The Linux kernel had strlcpy over 20 years ago. It was removed in favor of strscpy because the latter was judged a better interface. Here's a 2022 article: https://lwn.net/Articles/905777/

avadodin• 1 month ago

Returning an error is better but you're using ssize_t which is a tradeoff.

The race conditions appear to be a result of the Linux kernel implementation but UNIX style syscalls introduce these races by default. It is not an inherent flaw of the API or even the implementation Linux was using.

The only useable C string API has always been memcpy anyways.

senfiaj5• 1 month ago

I wonder, why not use a string buffer paired with its length? For example, maybe use struct that has char pointer, and 2 ints (occupied length + total buffer length). Almost like c++'s std::string. This null terminator thing really sucks, it's potentially insecure and often unperformant.

GalaxyNova• 1 month ago

Yes I have seen it happen a few times with `strlen` being called in a loop silently causing O(N) to turn to O(N^2)

senfiaj5• 1 month ago

Exactly, you can't write clean concise code when working with c strings. Almost every c string manipulation requires cognitive load: "Is the buffer size enough (including null terminator), should I reallocate it?", "I need to have the offset from the last concat, to make next concats performant", "Umm, shold I put null terminator at i or i + 1?"... It really sucks, it's akin to death by thousands of cuts.

jkrejcha3• 1 month ago

Reminds me of an article[1] that described how he cut GTA Online loading times by 70% because strlen was getting called for effectively every character in a string

[1] https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...

sgerenser• 1 month ago

Joel Spolsky coined the term “Shlemiel the Painter’s Algorithm” for this type of thing back in 2001: https://www.joelonsoftware.com/2001/12/11/back-to-basics/

chiph• 1 month ago

Pascal did/does this, but eventually someone wants a string longer than the size portion can handle. Or wants the number of characters not the number of bytes.

jerf2• 1 month ago

I wasn't a programmer in these days, so I don't know if there's some other major concern that would kill this, but I sometimes wonder about whether we could have / should have used variable-length integers. That is, something like, 0-127 byte strings get their length prefixed, 128 - 16383 get two bytes of prefix, and the probably-rare 16384 - 2097151 strings would end up with three, though proportionally by that point it's hardly anything. Or you could use the UTF-8 mechanism for packing the bytes, though that costs more and probably doesn't get anything we'd care about in the 1980s or 1990s.

It's a bit of extra code, yes. Not necessarily all that much, but some. On average it is only slightly more expensive than null termination, and considered as a proportion of the size of the strings themselves it's hardly anything. It's probably better than the strings getting hard-limited to 0-255, though, which was quite frequently a user-visible quirk.

Johanx64• 1 month ago

Dude, every sane language out there does this. Just generally with 4byte prefix. Null-terminated stuff has always been backwards compat stuff.

Pascal strings - historically and why people even remember this being an issue - were up to 255 chars in size, if not you had to use different string type.

You might still want raw pointers for all sorts of low level stuff, but you almost never want to have null-terminated strings for anything but back-compat, one of the worst things ever, even on memory constrained systems.

pjmlp9• 1 month ago

And then anyone that isn't stuck in 1976 will use open arrays.

none_to_remain2• 1 month ago

The size overhead of that is 2*sizeof(int) while the overhead of null termination is sizeof(char). If I remember the standard right, the former is worse by at least sizeof(char), and usually more in practice. This used to matter, sometimes still does.

senfiaj5• 1 month ago

Well, not saying to always use it, but if the string size is big enough, the overhead of 2 ints becomes relatively vanishing. For generic dynamically sized strings it probably has more advantages than disadvantages. But in any case, sure, if every single byte matters or some structure requires specific memory layout, then fine. I just don't think these things are the majority of use cases. Keep in mind that the cached lengths can increase performance, since you don't have to recalculate string lengths.

kgeist• 1 month ago

I would assume the difference is mostly negligible in practice due to the allocator rounding up the allocated memory size at least by the word size anyway (for alignment and simpler bookkeeping). You can also use variable-length encoding in the header to use 1 byte for most cases, similar to how UTF-8 does it: if the most significant bit is not set, we assume a 7-bit encoding, which can represent string lengths up to 127 using 1 byte, which is probably 99% of strings.

bnolsen2• 1 month ago

That's called a fat pointer. Null terminated c strings is the majority of memory errors out there.

WalterBright2• 1 month ago

Wonder no longer!

https://dlang.org/spec/arrays.html#dynamic-arrays

and

https://dlang.org/spec/arrays.html#strings

and for C:

https://digitalmars.com/articles/C-biggest-mistake.html

MBCook2• 1 month ago

A lot of them are strings coming from or going to user space right? So wouldn’t you have to do constant conversions?

maxlybbert• 1 month ago

It's definitely possible. And common, at least in some projects. The only real drawback is that sloppiness will lead to multiple slightly different nonstandard string types in the same project.

D-Coder• 1 month ago

Note that "360 Patches" is 360 uses of strncpy that have been removed, not necessarily bugs.

dpark• 1 month ago

I would imagine 360 patches removed way more than 360 uses of strncpy. But yeah, it’s not a given that each of these patches addressed a bug. (Also not a given that there were only 360 bugs fixed.)

lambdaone2• 1 month ago

This sort of boring grind is where the real work of systems engineering is done. Big infrastructure projects like this work on making the Linux kernel more reliable while still keeping it workable throughout the process move on the scale of decades, not months.

appplication• 1 month ago

On one hand I understand why it’s decade scale (the long tail of users/dependencies is really, really long) but on the other hand it doesn’t feel like a tenable pace at which we can make meaningful long term progress. Less of a gripe and I guess more a paradox of critical infrastructure.

devsda• 1 month ago

Did anybody else misunderstand the title as removing strncpy func for linux users ?

For a moment, I misunderstood it as (g)libc removing strncpy and was worried about the trouble its going to cause.

WalterBright2• 1 month ago

"The strncpy function within the Linux kernel has been a "persistent source of bugs" for years due to counter-intuitive semantics and behavior around NUL termination along with performance issues due to redundant zero-filling of the destination."

Huh. Whenever I've been asked to review C code, I always looked for strncpy and always found a bug with it.

qarl11• 1 month ago

Am I going to be the first person to ask this after five hours? Really?

Wouldn't this work be extremely easy to implement with an LLM coder?

qustio5• 1 month ago

I don't think the bottleneck was that it took six years to Ctrl-F strncpy and type in new code for each file.

qarl11• 1 month ago

It's a shame you're misrepresenting what is actually going on.

In another comment here I explained that I have run a test: asking Claude Code to add a substantial feature to 270 different C programs.

Despite your beliefs - it went extremely well.

qarl26• 1 month ago

I looked at the git history. The first three years were wasted waiting for a human to pick it up. He then very slowly submitted patches over 2 years.

Claude Code doesn't need to be interested to work.

Animats3• 1 month ago

This is a job for Claude!

What happens if you turn a job like that over to Claude Code? A mess? Good results? Code bloat? Worth trying on existing C programs.

qarl11• 1 month ago

I ran a test where I added a "light" mode to xscreensaver: unique changes to over 270 different C programs.

It mostly did an amazing job in a short period of time.

EDIT: Of course I get downvoted for saying this. HN isn't interested in reality any more.

ninjin• 1 month ago

Of course I get downvoted for saying this. HN isn't interested in reality any more.

I suspect that rather many of us are simply just tired of Claude and friends getting shoehorned into any conversation about programming at this point. It is about as fun as the Rust Brigade entering any discussion about C. It adds nothing new to the discussion and it is frankly tiring since we pretty much at any time have a handful of conversations on the front page already covering "AI" topics anyway (counting four at the time of writing this).

krupan4• 1 month ago

These stories with no details and no proof are not interesting or helpful.

jibal7• 1 month ago

The purpose of strncpy, which was originally part of the UNIX kernel code, was to copy file names to and from directory entries that consisted of a 2 byte inode number and a 14 byte zero-padded but not zero-terminated name field.

I started warning my colleagues against using it the moment I saw it for the first time about 50 years ago.

dare9445• 1 month ago

strncpy appears somewhere around the Unix v7 time frame, however only as function in the standard C library. It is not used in the v7 kernel itself.

jibal7• 1 month ago

The code for strncpy was in the UNIX kernel since at least V6. It was eventually added to the C library under the name strncpy. Sometimes those entries were processed in userland, e.g., by fsck. The utility of strncpy is noted in the C89 rationale (FWIW I was once a member of X3J11, the C89 standards committee):

"strncpy was initially introduced into the C library to deal with fixed-length name fields in structures such as directory entries. Such fields are not used in the same way as strings: the trailing null is unnecessary for a maximum-length field, and setting trailing bytes for shorter names to null assures efficient field-wise comparisons. strncpy is not by origin a "bounded strcpy," and the Committee has preferred to recognize existing practice rather than alter the function to better suit it to such use."

And I just found this comment from John Mashey (I never met John but he and I both worked under Ted Dolotta, John at Bell Labs and me at ISC in Santa Monica):

https://softwareengineering.stackexchange.com/questions/4380...

"I can answer definitively, since I wrote the originals ~1977, having moved from BTL Piscataway to Murray Hill. They were first named str*n, but were later renamed strn*, as there was some system in BTL that needed first 6 letters of external names to be unique.

I was working on kernel & user code that supported rudimentary per-process accounting, which started with someone else, but needed extensions due to big increase in UNIX systems in computer centers, who wanted more performance analysis. I.e. this was supported by commands like accton(1), acctcms(1),acctcom(1), acctmerge(1) (all in UNIX/TS 1.0, Nov 1978, which was ~Research V7 with first steps of PWB/UNIX influence. Think of that as 1.0, then PWB/UNIX 2.0, then UNIX System III...

The records described in acct(5) held the last 8 characters of the command pathname,truncated if necessary and thus possibly not null-terminated. I found multiple instances of inline code to manipulate these, which seemed a bad idea, so I wrote the str*n functions and replaced the inline code, and also used them in the various commands.

I also thought it was a good idea for better code safety.:-) Sigh."

JdeBP3• 1 month ago

In Unix Seventh Edition, ls and others read directory entries with fread() and parsed the struct direct themselves in application-mode code. The C library and application mode matter, here.

On the gripping hand, there is no strncpy in the Spinellis 7th Edition source code; 4.2BSD was using strncpy() inside readdir() in 1982, though.

twothreeone• 1 month ago

wow, very humbling. I'm actually amazed how many people contributed to this. It's easy to get attribution for "cool new features", but arguable removing bad features is even more important for something as fundamental as the kernel. Cudos!

I'm sure these are the sorts of things that will go down as folklore from the "founding ages", when everyone will have forgotten how to understand source code in 50 years and the Claude/Codex cruft just silently keeps piling on and burning the majority of our planets energy.

skywal_l• 1 month ago

Reminds me of Deepness in the Sky (Vernor Vinge) where a guy maintains a ship by doing software archeology. He is the only guy who knows what the Unix epoch is.

needusername• 24 days ago

Doesn‘t he hack the ship and confuse the Unix epoch with the moon landing?

nuc1e0n3• 1 month ago

I'm of the opinion AI slop code will become untenable way before that.

bigstrat20035• 1 month ago

I'm of the opinion it already is. But it seems that many people have yet to reach the point of being fed up with it.

zaik• 1 month ago

everyone will have forgotten how to understand source code in 50 years

I don't think this will happen. Human desire to understand how things work will still be around in 50 years.

DerSaidin• 1 month ago

strtomem_pad seems redundant with memcpy_and_pad, and also it requires the preprocessor: https://github.com/torvalds/linux/blob/1a3746ccbb0a97bed3c06...

I was curious: Why have it, instead of just using memcpy_and_pad?

AI's answer (paraphrased) was * Avoid possible bugs from manually write sizeof(dest) * Enforces the __nonstring Attribute * signals: "I am converting an actual C-string into a fixed-width legacy memory field." vs copy binary data & pad it.

Interesting to learn about the __nonstring attribute:

https://github.com/torvalds/linux/blob/1a3746ccbb0a97bed3c06... https://github.com/search?q=repo%3Atorvalds%2Flinux+__nonstr...

cm21872• 1 month ago

A lot of pain and suffering to avoid having a string datatype.

edoceo• 1 month ago

What's a way they could get a strong data type here? Wouldn't that also require a large refactor of the code around strncpy to use the type and its functions?

cm21872• 1 month ago

Today yes, but 40 years ago someone made the decision that a string was a char array and that every string manipulation going forward would require manipulating arrays. Talking about costly decisions.

It’s actually interesting to compare the pain and suffering of switching to a string datatype in the 80s (refactoring the limited code base then) vs the next 40 years of unnecessary boiler plate syntax and bugs for not having this type in key APIs.

teo_zero• 1 month ago

A lot of pain and suffering to avoid having a string datatype

No, a lot of pain and suffering to work around the lack of a string datatype in C.

sirwhinesalot• 1 month ago

I have in the past made fun of the Linux kernel devs, supposedly some of the best C developers in the world, for not knowing how to make stringbuffer and stringview types, but to be fair to them we didn't have the consensus we have today on the topic.

You know who did have the right idea though? Dennis Ritchie, who proposed a fat pointer type for C all the way back in 1990. Would have made for a perfect addition to C99. Imagine how different the world might have been had the committee added that in.

We had a second chance with the release of the "C's greatest mistake" blog article from Walter Bright in 2007, essentially pushing for the same idea as Ritchie (slices/stringviews) but explained with much clearer language.

Alas, didn't make it to C11.

We're now in C23, still nothing. But we did get _Generic and VLAs! Party hard.

pjmlp9• 1 month ago

That only goes to show where WG14 priorities are.

anal_reactor• 1 month ago

but to be fair to them we didn't have the consensus we have today on the topic.

This is my pet peeve of teamwork. We can choose solutions A, B or C. Each has upsides and downsides. We debate for two weeks, then we choose nothing.

david-gpu• 1 month ago

Isn't that due to a lack of leadership, rather than a problem with teamwork itself? Somebody has to be ultimately in charge and willing to put a stop to endless debate.

flohofwoe4• 1 month ago

But we did get _Generic and VLAs! Party hard.

VLA has been demoted to an optional feature in C11 (good).

IMHO the current main problem is that the C stdlib is stuck in the K&R era and the stdlib APIs haven't even been updated to the language features added in C99 (e.g. make use of struct args and return values). A range struct (ptr/size pair) in the stdlib and new or updated string functions to use such ranges would already go a long way.

VorpalWay• 1 month ago

IMHO the current main problem is that the C stdlib is stuck in the K&R era and the stdlib APIs haven't even been updated to the language features added in C99 (e.g. make use of struct args and return values).

C++ has the same issue (only with more chaos and bloat). They add some new good idea (like optional) but don't update the rest of the standard library to make use of it. And they can't really, without breaking backwards compatibility.

I think looking at the edition system in Rust could be useful for C and C++ to start to solve this. Something like "If this source file has this pragma in it, compile that code with a new edition". It would have to be granular, per expression really, to handle macros (which is how it works in Rust too). What would it change in C/C++? Name resolution, you could get a different set of resolvable overloads, depending on which edition is active in the caller context. Not unlike an enable_if.

The same would work in C: depending on the caller edition, expose function signatures.

ethbr1• 1 month ago

"C's greatest mistake" blog article from Walter Bright in 2007

https://digitalmars.com/articles/C-biggest-mistake.html

And because it came up in my search and the bikeshedding discussion made me chuckle, reddit on same: https://www.reddit.com/r/C_Programming/comments/90uq7c/cs_bi...

Am curious about this esoterica, if anyone can confirm/deny:

> Speaking of [C] arrays decaying into pointers, does anyone know why this behaviour was designed in the first place?
> It was so that B code could be compiled as C with minimal changes. The designer felt that this would encourage people to switch from B to C. In B an array declaration actually defined a pointer and an array, with the pointer initialized to point to the array's first element.

fulafel• 1 month ago

Link to Ritchie's proposal: https://web.archive.org/web/20150611114358/https://www.bell-...

HackerThemAll• 1 month ago

I have used the DJB's stralloc library a lot. Not a single vulnerability, binary string safe. Good enough for 99% of applications.

https://cr.yp.to/lib/stralloc.html

pjmlp9• 1 month ago

Now lets put that work into money, to assert what was the cost impact of replacing strncpy().

rswail20• 1 month ago

Things that have bugged me for 40 years...

* NUL terminated strings (and now, non UTF-8 encoded strings on input/output)

* Using LF or CR or CRLF as line terminators, and pipe/comma-delimited fields when there were other unambiguous ASCII characters that could have been used (eg, GS, FS, RS) that would have made the encoding/decoding of line termination an I/O thing keeping HT/VT/CR/LF/FF as literally print related codes.

flohofwoe4• 1 month ago

non UTF-8 encoded strings on input/output

UTF-8 on stdin/stdout works perfectly fine (unless you are on Windows of course, which is stuck in in the early 90s when it comes to international text encoding).

Using LF or CR or CRLF as line terminators

This is also an operating system convention, and it would be better if programming languages wouldn't try to "guess" the correct line endings, since this causes more problems than it solves - but again, this is mostly a Windows specific problem, and it's Microsoft's job to finally bring Windows into the current century.

rswail20• 1 month ago

No, it was an Apple, Unix, and Microsoft problem.

Unix used LF, Apple used CR, Microsoft used CRLF.

They are all ASCII carriage movement codes, which is about driving the paper feed and print head of an ASR-33 or equivalent.

So they all made the "wrong" decision about what to store in a file.

They just chose different wrong characters.

skywhopper• 1 month ago

What programming languages try to guess line endings? Or are even aware of them?

EvanAnderson2• 1 month ago

I did a project to translate data framed in the ASCII field/record separator characters and it was gloriously easy. All the ugly escaping considerations with comma-delimited data went away and it became much easier.

ptx• 1 month ago

What happens when the data contains the record or field separator characters?

I suppose you could document that it's unsupported, and just drop or reject such values, but then the system couldn't be used to handle test data for such systems, for example.

Parodper9• 1 month ago

LF makes the most sense, but they're all fine for text files. The issue is that CSV isn't text.

Last time I had to handle CSV files in bash, I converted them internally to RS and FS.

Ekaros• 1 month ago

Line feed resetting position really makes no sense. It should just continue text from where the cursor was but on next line. Like staircase. You need CR to go back to start.

codedokode2• 1 month ago

non UTF-8 encoded strings on input/output

I would just use UTF-8 everywhere.

rswail20• 1 month ago

Storing them as 32 bits wide in memory means you can at least index by a codepoint (if not a glyph).

brewmarche• 1 month ago

Now with Unicode we actually have even more:

NL Next line (from EBCDIC?)

LS Line separator (invented by Unicode)

PS Paragraph separator (same)

The Unicode standard says that in addition to CR, LF, CRLF and the above, vertical tabs and form feeds should also be treated as line separators.

rswail20• 1 month ago

In all the comments in this thread it's interesting how people confuse:

* NUL: An ASCII non-printing character with the byte value of 0

* NULL: A pointer that does not point to usable memory with the value that compiles in C to be equal to ((void *) 0).

layer85• 1 month ago

NUL was always just an abbreviation for null: https://www.rfc-editor.org/rfc/rfc20.html#section-4

I don’t think anyone in this thread is confusing the null character with the null pointer.

rswail20• 1 month ago

I've seen a lot of confusion, where people are talking about checking for a NULL at the end of a list of pointers which is very different to a NUL at the end of a string.

Yes it was an abbreviation in ASCII, as are all the non-printable first 32 codes.

kstenerud• 1 month ago

strncpy is 99.999% of the time NOT the correct function to call, so this is a huge win.

It's just a shame that such a confusing name was chosen for such a niche use case (fixed width records that require null padding).

stcg• 1 month ago

I wonder what is the difficulty in rewriting strncpy uses that makes it take six years? Was it widespread? Or was it more of a long going effort, where it was only changed if there were some changes in the same file? Or is there some other thing that makes it difficult?

GTP2• 1 month ago

I always thought that srncpy was the safe alternative to strcpy. Now that I think of it, I'm unsure if the NUL terminator is counted into strncpy's size or not, which would be a likely source of errors. But, could someone explain better what the problems were? And also, would have to pick the right function in the list of given alternatives much better?

GabrielTFS• 1 month ago

The issue with strncpy is that it doesn't actually necessarily terminate - in fact in any case where the source is larger than the destination it will just leave it unterminated (like, it will copy the last character it can from the source instead of terminating the destination string with a NUL)

rurban2• 1 month ago

No, the safe alternatives end with _s. They do check matching buffer sizes, and enforce zero-termination. Unfortunately WG14 hates them also, because Microsoft. Microsoft did indeed break some of the, but you can use better alternatives, like my safeclib

GTP2• 1 month ago

No, the safe alternatives end with _s.

Could you please elaborate on this? Both `man strncpy_s` and `man strcpy_s` didn't return any manual page on my Linux system.

thiht• 1 month ago

In place of strncpy, Linux kernel code should use strscpy() for NUL terminated destinations, strscpy_pad() for NUl-terminated destinations with zero-padding, strtomem_pad() for non-NUL-terminated fixed-width fields, memcpy_and_pad() for bounded copies with explicit padding, or memcpy() for known-length memory copies

What a nightmare, does it have to be so convoluted?

MarkMarine• 1 month ago

Performance. A safe Swiss Army knife function that did most of this would be slow because of the internal branching you’d need to be safe, and because there is developer intent in the selection of these functions. I’d rather have the choice and clear dev intent when I see the function used when reading code.

hahn-kev• 1 month ago

Couldn't they at least give them better names?

bobmcnamara4• 1 month ago

does it have to be so convoluted?

Getting strncpy right always has been

henrypoydar• 1 month ago

No code is faster than no code.

Linux eliminates the strncpy API after six years of work, 360 patches

Related Stories