5 minutes of coding yields a 6%+ boost to Linux I/O performance

https://www.phoronix.com/news/Linux-Caching-Time-Block-IO

by mfiguiere • 3 years ago

128 37 3 years ago

How many hours, days, weeks of thinking?

Axboe shared about his latest interesting Linux I/O performance optimization, "Something I've had in the back of my mind for years, and finally did it today. Which is kind of sad, since it was literally a 5 min job, yielding a more than 6% improvement. Would likely be even larger on a full scale distro style kernel config."

Seems irrelevant. From the sound of it this was something that would always have been easy for him to do. He just never got to do it until now.

nomel• 3 years ago

Even in open source software, the eyes of many will never come close to the brains of one (for better or worse).

sophacles• 3 years ago

Maybe, maybe not. Lots of stuff gets put in "the back of my mind" because I'm not sure how to do it immediately. Maybe I'll think about it occasionally, or note a consequence like "oh this would be affected if I get around to X". Then when I actually sit down to do it weeks or months later it seams easy - even though when I first thought it I wasn't sure how to even do it.

Point being- it's pretty unlikely that something bouncing around the back of someone's head for years would have taken 5 minutes to do right away, just as it's unlikely that it would have taken 6 years if they'd started when they first had the thought. I suspect there's been some amount of thinking, and potentially that other work was subtly influenced by this thought so that finally doing the task was "easy".

droningparrot• 3 years ago

It may have taken 5 minutes to write the code, but it sounds like they spent years thinking about the change to make.

readyplayernull• 3 years ago

That's what I always tell at work.

cozzyd• 3 years ago

It takes less than 5 minutes to write out the Standard Model Lagrangian too :)

mr_mitm• 3 years ago

Pretty sure I can write down a theory of everything as X=0 where X is a suitably defined mathematical object.

blibble• 3 years ago

the amount of codebases I've worked on that spend most of their CPU time measuring the time is not inconsequential

jl6• 3 years ago

I am reminded of the story of the plumber who was called to fix a water leak. He replaced a washer and charged $100. The client was outraged: “That washer was only a dollar’s worth of parts!”

The plumber replied: “Sure. I’m charging you $1 for the washer and $99 for knowing where to fit it.”

andersource• 3 years ago

Sounds like a variation of the Ford-GE $10k chalk mark story:

https://www.reddit.com/r/todayilearned/comments/2b4n7a/til_h...

dublin• 3 years ago

This famous "granddaddy of them all" story about knowing where to make a chalk mark was validated in a 1965 letter to the editor of Life magazine. Although most Internet sources act as though this story is apocryphal, it has some real (if not conclusive) provenance, and definitely fits the known personality quirks of both Henry Ford and Charles Steinmetz...

slowmovintarget• 3 years ago

More generic $50K version: https://www.et.byu.edu/~tom/jokes/Consultant_Engineer.html

bitwize• 3 years ago

I heard the same story except it was a DEC technician. "Hitting the machine: $1. Knowing where to hit it: $999."

GiorgioG• 3 years ago

Everyone knows that programming is just typing.

thebruce87m• 3 years ago

Now I’ve got to copy and paste out of the ChatGPT window too. That’s two extra steps.

pjmlp• 3 years ago

The problem will be when ChatGPT types it in for us.

reportgunner• 3 years ago

It's you who makes it harder for yourself.

zwieback2• 3 years ago

Title should be "Six years of thinking allow programmer to solve problem in 5 minutes".

I remember contracting for IBM in the 90s and we had a huge performance boost due to caching and, like the anecdote in this post, there was a slight chance of something going out of date. The IBM guys said "no thanks, IBM prefers bullet proof over fast".

toasted-subs• 3 years ago

Then ibm got outpaced by the rest of industry.

Bulletproof typically means rigid in design. Trading one bag of worms for another.

zwieback2• 3 years ago

Yep, especially as IT moved downmarket from mainframes run by government, military and large corporations to the average Joe, who can reboot, reinstall or just buy a new one.

leeoniya• 3 years ago

Title should be "Six years of thinking allow programmer to solve problem in 5 minutes".

"You didn't pay me $1k to press three buttons; you paid me $1k for knowing which three buttons to press."

EDIT: oh, i see a version of this was already posted in another comment :)

HocusLocus• 3 years ago

(c.1986) Telephone company accounting system with System/38 (pre AS/400) and System R (early SQL), CL+RPGIII. The call rating process required half a dozen lookups to other tables to complete each record in a batch. Something that would be well handled by a modern join. But at the time the vendor was like, "Gee Ma it works! Ship it." Compiled RPG III was lightning fast and a lot like working in assembly with fixed/prealloc string handling. I made a template and adapted it for each table that cached recent lookups for each. 6x improvement. Then realized that since records were autonomous and batches run and posted separately, it was never necessary to process calls in chrono order. Pre-indexing on destination number resulted in cache hits almost all the time and 10x speed improvement over the 6x.

When I asked my Dad how long something took, he'd say something like "2 years and 20 minutes." 2 years to get around to it. Mom took his habit into the real world and published an ad for the company with a cut out "Round Tuit" and copy, "How many times have you heard someone say, 'I'll do that when I get a round tuit.' Now that you have your very own Round Tuit, there's no limit to what you can accomplish!"

Round Tuits were cut out and appeared on walls all over town.

winrid• 3 years ago

I think when I was 17 some old guy asked me why I hadn't changed my oil yet* and I said I hadn't got around to it. He ran into his shed and came out with a round piece of wood that said "to it" on it... So I changed the oil in his driveway.

* The oil was still <5k mi old. People are funny.

drittich• 3 years ago

And let's be honest, most of those 6 years was figuring out what to name the variables.

harshaw• 3 years ago

this is a pretty standard performance win to avoid constantly getting the time when you don't need it. Certainly seen this win at work (AWS) but also likely in various other projects repeatedly. But bias for action for getting it done!

loeg• 3 years ago

Yeah. We use and cache rdtsc a fair amount to avoid its relatively high overhead in the storage stack I'm working on as well.

vlovich123• 3 years ago

I wonder if there’s many other spots where time is being read multiple times instead of reading it once and caching it. All that being said, it’s also surprising how relatively expensive reading time is. Even rdtsc is 50-100ns if I recall correctly, but to put that in perspective that’s basically the performance of a 10-20Mhz chip to do an operation that should be a simple register read of a latch that’s incremented on every clock cycle. yes yes - I know in reality it’s a heck of a lot more complex for lots of reasons, but still, given how frequently we want to time things or obtain a timestamp, it’s surprising to me we don’t have a faster instruction and we have to do a bunch of caching (eg you’d not likely bother caching just the addition of two variables because addition is so fast)

lelanthran• 3 years ago

. All that being said, it’s also surprising how relatively expensive reading time is.

Typically you don't need granularity as fine as you think you do. Reading time at the finest granularity available is going to kill your throughput/latency no matter what because you are probing the hardware, so make sure you need that very fine granularity.

IME, most (99.999%) of the case you need the time, you can get by with coarse granularity. I worked on realtime systems to control a jig in a factory, using embedded 486-class processors, with hard-realtime kernel patches, and even then I had no need to use the non-coarse timers when recording events for observability.

In any case, here's the relative costs of the different timers: https://www.lelanthran.com/chap5/content.html

Dwedit• 3 years ago

Meanwhile on Win32, GetTickCount performs no system calls at all, simply reading a value from a shared memory page. The operating system is updating that value behind the scenes.

gertop• 3 years ago

That's also how it worked on the commodore 64 in the 80s.

wtallis• 3 years ago

This isn't about performing a syscall to get the time, this is about the kernel handling a syscall and in the process checking the time too many times.

And getting the time without a full syscall is hardly unique to Windows. On Linux, that's almost the entire point of the vDSO.

vrmiguel• 3 years ago

Isn't that the same idea as Linux's vdso?

https://man7.org/linux/man-pages/man7/vdso.7.html

misiek08• 3 years ago

A giant ship’s engine failed. The ship’s owners tried one ‘professional’ after another but none of them could figure out how to fix the broken engine.

Then they brought in a man who had been fixing ships since he was young. He carried a large bag of tools with him and when he arrived immediately went to work. He inspected the engine very carefully, top to bottom.

Two of the ship’s owners were there watching this man, hoping he would know what to do. After looking things over, the old man reached into his bag and pulled out a small hammer. He gently tapped something. Instantly, the engine lurched into life. He carefully put his hammer away and the engine was fixed!!!

A week later, the owners received an invoice from the old man for $10,000.

What?! the owners exclaimed. “He hardly did anything..!!!”.

So they wrote to the man; “Please send us an itemised invoice.”

The man sent an invoice that read:

Tapping with a hammer………………….. $2.00

Knowing where to tap…………………….. $9,998.00

Effort is important but experience and knowing where to direct that effort makes all the difference.

dirty C&P from: https://medium.com/@oceanbcreative/the-ship-repair-man-story...

robbiep• 3 years ago

The really awful thing about the story you’ve reposted is it’s actually a real story about a really intriguing and amazing person. Why tell the moral of the story when you can tell the real story?

https://en.m.wikipedia.org/wiki/Charles_Proteus_Steinmetz

https://www.smithsonianmag.com/history/charles-proteus-stein...

5 minutes of coding yields a 6%+ boost to Linux I/O performance

Related Stories