Where Linux's load average comes from in the kernel

https://utcc.utoronto.ca/~cks/space/blog/linux/LoadAverageWhereFrom

by zdw • 4 years ago

172 38 4 years ago

utcc.utoronto.ca

tanelpoder8• 4 years ago

When I was looking into this, I found that just running ps (with -L option to see all threads, not just processes/thread group leaders) with some grep/sort/uniq was the easiest way to break down where does the “too high Linux system load” come from. No need to compile C code or have root access to drill down into load. And you could drill down further by sampling some additional /proc/PID/task/TID/ files, like “syscall”, “stack” to see which (blocked) syscall is contributing to the load and where in the kernel is it stuck. Knowing what kind of process/thread level /proc files are available and reading/sampling them with a shell one-liner is a powerful entry point for performance troubleshooting and may allow you to delay writing advanced kernel tracing scripts further.

For those that are interested:

https://tanelpoder.com/posts/high-system-load-low-cpu-utiliz...

anonymousDan• 4 years ago

Interesting link thanks. The part I always find tricky with kernel debugging is distinguishing normal vs abnormal behaviour (e.g. how many kworker threads is too many)?

tanelpoder8• 4 years ago

Yep, one thing that helps in this context is to only look into threads in R or D state (and ignore the hundreds or sometimes thousands of kernel threads that are just sleeping or waiting for more work to do in idle (I) state).

But yeah, even that doesn't clearly tell you what amount of active threads is normal for your workload/configuration. Or you can just save the thread count info (grouped by thread state) and save/graph it over time, to have a better idea of what's normal. The difference of this approach vs just graphing Linux-reported load is that you'll get a breakdown of how much of it was CPU demand vs sync I/O demand. Or you could even break the the thread states down further, by syscall and WCHAN, for example. This will give you a much more detailed idea of why the load is high, not just that it is high.

(Like I mentioned in the article), I even have a tool for sampling /proc at regular intervals (1 Hz by default) and saving it into hourly CSV files, so you can easily go "back in time" and zoom into short spikes and see what was going on:

  $ xcapture
  
  0xTools xcapture v1.0 by Tanel Poder [https://0x.tools]
  
  Sampling /proc...
  
  DATE       TIME             PID     TID USERNAME   ST COMMAND        SYSCALL    WCHAN                    
  2020-10-17 12:01:50.583    6404    7524 mysql      R  (mysqld)       fsync      wait_on_page_bit          
  2020-10-17 12:01:50.583    6404    8944 mysql      D  (mysqld)       fsync      wait_on_page_bit          
  2020-10-17 12:01:50.583    6404    8946 mysql      D  (mysqld)       fsync      wait_on_page_bit          
  2020-10-17 12:01:50.583    6404   76046 mysql      D  (mysqld)       fsync      wait_on_page_bit          
  2020-10-17 12:01:50.583    6404   76811 mysql      D  (mysqld)       fdatasync  xfs_log_force_lsn         
  2020-10-17 12:01:50.583    6404   76815 mysql      D  (mysqld)       fsync      blkdev_issue_flush        
  2020-10-17 12:01:50.583    8803    8803 root       R  (md10_resync)  [running]  0                         

  DATE       TIME             PID     TID USERNAME   ST COMMAND        SYSCALL    WCHAN                    
  2020-10-17 12:01:51.623    6404    7521 mysql      D  (mysqld)       pwrite64   xfs_file_buffered_aio_write 
  2020-10-17 12:01:51.623    6404    7524 mysql      D  (mysqld)       fsync      xfs_log_force_lsn         
  ...

bragr2• 4 years ago

One of my old Linux sys admin interview questions was describing a situation with a high load number but low CPU utilization. People who said that was impossible or similar got shown the door and people who knew or could reason their way to IO wait/problem would get to proceed.

tanelpoder8• 4 years ago

There's also one interesting addition that people may not be aware of: Synchronous I/O that blocks (like pread64/pwrite64) will contribute to Linux system load (threads in D state).

Asynchronous I/O completion checks (libaio's io_getevents) that are willing to wait for I/O completion, will not contribute to Linux system load (threads in S state).

Asynchronous I/O submissions (libaio's io_submit) either quickly submit their I/O (a small amount of time in R state) OR get stuck in io_submit() if the underlying block device I/O queue is full. When io_submit() gets stuck, then you're sleeping in D mode, thus contributing to system load again.

walrus01• 4 years ago

one possible 'correct' answer for something like this would be iowait caused by slow disk performance on a degraded raid array that was still operational.

as one of many possible theoretical scenarios.

shon• 4 years ago

We used to ask the same question. TBH I haven’t read the f’ing article yet but… I’d summarize this as load=Run-que length. Number of run-eligible processes blocked by something where something is usually IO in a low CPU utilization scenario.

EdSchouten• 4 years ago

The fact that the load average also counts threads stuck on I/O happens to be Linux specific. On the BSDs it only measures CPU utilization.

So it may be the case that you showed people the door, simply because their experience was based on non-Linux operating systems.

bragr2• 4 years ago

One does typically prefer Linux experience for a _Linux_ sys admin, but don't get me wrong that's just one of many screening question. That said, if someone doesn't have enough Linux experience to know how Linux differs from *BSD or Solaris or whatever they have time on, that seems like a valid exclusion to me. I don't tend to put any stock of claims of being able to be a quick study unless they've obviously done some interview prep on the subjects they're unfamiliar with. Best way to show you are a motivated, quick study is by being a motivated, quick study.

belter2• 4 years ago

"The many load averages of Unix(es)"

https://utcc.utoronto.ca/~cks/space/blog/unix/ManyLoadAverag...

lamontcg• 4 years ago

processes stuck in a D state on stale NFS mounts.

pram• 4 years ago

Ah my old nemesis, you just had to reboot the machine to unfuck it lol

Drybones• 4 years ago

Recently, I was asked an interview question about what affects Linux IO load averages with a MANGA company. When I mentioned that one of the multiple possible reasons could be IO, he acted like I said the sky was green. He did not push me through the process. He was a systems engineer too and not just some HR person reading a script.

belter2• 4 years ago

From the code comments...

* kernel/sched/loadavg.c

* This file contains the magic bits required to compute the global loadavg

* figure. Its a silly number but people think its important. We go through

* great pains to make it work on big machines and tickless kernels.

R0b0t1• 4 years ago

The comment really undersells it. The fact that it is a regular number, despite not being strongly tied to something real, does make it useful. Even when you have a lot of IO wait causing an extremely large number you are being told something useful.

waynesonfire• 4 years ago

If I understand the authors use-case to get more refined top results, from the first paragraph,

"Suppose, not hypothetically, that you have a machine that periodically has its load average briefly soar to relatively absurd levels for no obvious reason; the machine is normally at, say, 0.5 load average but briefly spikes to 10 or 15 a number of times a day. "

$ sudo perf sched record -- sleep 5 && sudo perf sched latency

seems to do the trick. I'm not even a perf engineer just did a simple google search. Though, that makes for pretty crummy blog content.

tanelpoder8• 4 years ago

Perf sched shows you scheduling latency (time spent waiting on CPU runqueue in Runnable state), but not "demand" for (some of) the system resources like the load figure tries to estimate.

I think the author didn't realize that the unit of load is just "number of threads" doing _something_ (wanting to be on CPU, running on CPU or waiting for I/O in D state on Linux, just CPU stuff on other Unixes).

Load average is just "average number of threads" doing something over last 1,5,15 minutes.

So if just the single-number averaged over multiple minutes is not good enough for drilling down into your load spikes, then you just go look into the data source (not necessarily source code) yourself. Just use ps or /proc filesystem to list the _number of threads_ that are currently either in R or D state. That's your system load at the current moment. If you want some summary/average over 10 seconds, run the same commant 100x in a row (and sleep a bit in the between) and count all threads in R & D state (and then divide by 100 to normalize it to an average).

It's basically sampling-based profiling of Linux thread states.

wbh1• 4 years ago

Similar article from Brendan Gregg a few years back: https://www.brendangregg.com/blog/2017-08-08/linux-load-aver...

tie_• 4 years ago

Install atop and configure it to sample every second.

I can't count the number of times this has helped solve a mysterious behavior. Atop is king.

geocrasher• 4 years ago

Yes, atop is great for this! There's also "sar" (system activity report) which can do similar things. Both are quite helpful.

jeffbee• 4 years ago

It's great but it's very expensive so be aware that if you are down to the level of caring about things in the single-digit percentages of your budget, something like atop once per second could easily consume that amount of resources. A lot of these stats are really not cheap at all. E.g. you may find a substantial difference in the amount of CPU time needed to produce /proc/pid/stat as opposed to /proc/pid/statm. And some of the system-wide stats are even worse.

winrid• 4 years ago

Be careful running atop with the -R flag on machines that create lots of short lived threads or do lots of IO, like many databases.

nousermane• 4 years ago

Pretty sure that running "top -b >/var/log/whatever" in background would catch the culprit(s) of originally reported "spikes".

the8472• 4 years ago

There is a newer alternative, pressure stall information[0], split by CPU, memory and IO pressure. htop can display the rolling PSI averages and one can also see the total time during which some tasks were stalled. They're also available per cgroup.

[0] https://lwn.net/Articles/759781/

mardifoufs2• 4 years ago

Now that makes me wonder how does Windows deal with this? Say when you open the task manager, what does the cpu usage % actually mean? I'd guess it's not measuring it in the same way Linux does, but I'd be surprised if it used pressure stall information?

SuperQue• 4 years ago

Windows users just look at CPU utilization, which is what most Linux users should be doing as well. Monitoring both total system and single core utilization.

Of course, 100% CPU use also isn't a "problem", it's just a fact. PSI is the thing that actually reveals problems. I'm not aware of a Windows equivalent.

tanelpoder8• 4 years ago

By the way, PSI has the same shortcoming of not accounting the time of threads waiting for async I/O completion as I/O pressure.

zaarn3• 4 years ago

Yes, but async I/O means the process can continue doing other, potentially useful, work, while waiting for async I/O to complete. So if a process has issues an async I/O, it's not blocked or dead, like a process who issued sync I/O.

And on the topside, if you block on the io_submit and friends, async I/O counts towards the PSI and load average again. Ie, when you can't do useful work because the device queue is full, for example.

So it's not as much of a shortcoming as you think, because if the system is I/O exhausted, async I/O processes will quickly contribute to load as well once the I/O queues are nearly full or full.

Where Linux's load average comes from in the kernel

Related Stories