Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

http://www.linuxatemyram.com/

by yankcrime • 15 years ago

155 70 15 years ago

eliben• 15 years ago

The title of the link (unlike the linked page) suggests that Linux is somehow to blame here... Why? Using RAM for disk caching is a good thing.

Besides, this technique is employed by Windows as well. It's just that the average Windows user probably doesn't know how much free RAM his PC has (well, the really average user doesn't know how much total RAM it has). My Windows 7 box takes at least 2GB of memory to itself for disk caching, AFAIU.

yankcrime• 15 years ago

My apologies, that's certainly not what I meant to imply (and hence the use of the word seemingly). The point I was coming from was with regards to the number of times I've had to explain these concepts to an exasperated friend or cow-orker, when they've exclaimed something along the lines of "I've just looked at `top` and there's no free memory!".

roc• 15 years ago

Windows started doing disk caching with Vista and users certainly noticed and complained in much the same way.

barrkel5• 15 years ago

Windows didn't start doing disk caching with Vista. Perhaps it used memory more aggressively, or changed the way it did the accounting, but it certainly didn't start doing disk caching just with Vista, as anyone who remembers the difference in performance of DOS before and after running SMARTDRV will attest.

halfasleep• 15 years ago

Windows started doing pre-emptive caching with Vista, based on the applications you tend to run.

Luyt• 15 years ago

I think the average user doesn't even know what RAM is. My neighbour, for example, doesn't know the distinction between the space available in RAM and the capacity of his harddisk. Which led to a strange misunderstanding when he told me his PC had '250 giga of internal memory'. I tried to explain than with 'internal memory' people usually mean something else, but I didn't think he understood. Never mind, he doesn't grasp hierarchical filesystems either, but all is fine as long as he can edit his photos (which in his perception are kept within Elements) and 'surf the internet'.

cullend• 15 years ago

So basically the same thing Vista was doing (prefetch, and admittedly it did a shitty job with >1GB ram), except since its Linux, Oh its all fine guys, we won't destroy the OS' reputation.

bbatsell• 15 years ago

Right, the only thing wrong with Windows Vista was its RAM caching strategy.

recoiledsnake3• 15 years ago

Maybe not, but there were a lot of people needlessly going bonkers over the disk light blinking (SuperFetch) and free memory numbers in the task manager(exactly the same problem that the referenced site is trying to explain).

See http://www.codinghorror.com/blog/2006/09/why-does-vista-use-...

simias• 15 years ago

The page is about explaining a correct behavior, not justifying some bogus memory usage. There's no reputation at stake. All modern operating systems are expected to behave that way, free RAM is wasted RAM.

samdk• 15 years ago

Linux has been doing this for a lot longer than Windows has. (Which I only actually know because there was so much FUD about Vista's pre-fetching when it came out.)

edit: Sorry, getting prefetching and SuperFetch confused.

gaius• 15 years ago

Unix has been doing it... You talk as if Linus personally invented the idea of caching.

recoiledsnake3• 15 years ago

Doing what? Prefetching? It has been there since XP.It is not just simple caching. Windows probably was doing caching even before Linux existed.

What Vista does is SuperFetch and I don't think it's there in Linux even now, except that someone wrote a program that would do a similar thing but it's not really used that much.

barrkel5• 15 years ago

MS shipped SMARTDRV with DOS 4.01, which shipped in 1989. By my reckoning, Windows (running on DOS) has had disk caching since before Linux existed.

michaelcampbell• 15 years ago

I dare say Vista did a plenty good job of destroying its reputation without anyone batting an eyelash at this.

recoiledsnake3• 15 years ago

Prefetching is not the same as caching. Vista's SuperFetch is even more different.

pixelbeat• 15 years ago

Heh, nice site.

Note I find this useful for showing how much RAM apps really are using: http://www.pixelbeat.org/scripts/ps_mem.py

StavrosK• 15 years ago

A good way I read somewhere was "Unused RAM is wasted RAM. Linux uses it to speed up your disk and gives it back to programs when they need it".

slavak2• 15 years ago

Wouldn't it make more sense to just include the RAM used for disk-caching when calculating free memory? After all, the whole reason it's being used for disk-caching in the first place is that it's free! The Linuxoids that know what this even means can always explicitly look for the line that says how much memory is being used for caching...

sp332• 15 years ago

The Windows task manager has a separate field called "Available". Right now I have 8190 MB total, 3627 cached, 4151 Available, and 537 free (not sure if that adds up perfectly, it was changing while I typed it).

masklinn2• 15 years ago

OSX's activity monitor does something similar, it has 4 statuses: Wired, Active, Inactive and Free.

Wired and Active are basically the RAM being used (Wired is a special activity status for stuff which can not be moved to disk), Inactive is the cache (generally programs which were used recently) and Free is… well, free.

When checking used/free, Inactive and Free count towards free, Wired and Active towards used (though that's not always consistent).

Also, I don't think you have 8TB RAM.

barrkel5• 15 years ago

It costs time to zero pages; disk cache pages are not zeroed. Not having a source of zero pages will add latency to large memory allocation requests.

(The pages must be zeroed to avoid the security problem of snooping on data you might not have access to.)

slavak2• 15 years ago

Exactly the kind of thing users not familiar with the way Linux handles memory DON'T consider when they check how much free memory they have. If you're the sort of person concerned with the overhead of zeroing pages then you can look at more detailed memory-usage info to find the details you're looking for.

colanderman3• 15 years ago

This is misleading, as the kernel will often swap out processes rather than shrink the disk cache. Its tendency to do this is controlled by the vm.swappiness parameter, which by default heavily favors swapping: http://kerneltrap.org/node/3000

Decreasing this value will favor reducing the cache size and will tend to increase responsiveness. You can set it in the /etc/sysctl.conf file. Low values (≤ 5) are especially good for SSDs where you want to minimize swap usage.

kindlyviking• 15 years ago

The article mentions this on the second page at the bottom: http://www.linuxatemyram.com/play.html

colanderman3• 15 years ago

"But look you found the notice didn't you?"

"Yes," said Arthur, "yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of The Leopard'."

jerf5• 15 years ago

I often just turn swap off entirely. A normal workload on my 4GB linux machine will frequently fail to fill even 2GB of buffers. At the moment my machine has still managed to fail to fill 800MB of my RAM with anything and I'm running two virtual machines and doing my usual work load.

I do have a swap partition but only so I can hibernate.

requinot594• 15 years ago

> I often just turn swap off entirely.

And you're smart to do so. Swap is useless for 99% of end-user systems. ChromeOS doesn't use any swap partition, for instance.

As for hibernation, it's not available if you use a crypted swap, which is wise to use (at least on a laptop).

As a result, I, too, disabled swap completely.

nwmcsween2• 15 years ago

Swap is mainly an emergency backing, i have nagios message me whenever swap hits x amount depending on the server, it's better than oom killer picking off processes.

stcredzero3• 15 years ago

Is there a way to decrease the swappiness of OS X?

jacques_chester• 15 years ago

Low swappiness is also useful in tightly controlled VPSes. Swap is absolute murder on a VPS platform. I leave it on as emergency backstop, but otherwise I have turned down to 0. I simply do not want processes swapped out, ever.

wzdd• 15 years ago

This paragraph: 'But what do you call memory that is both used for something and available for applications? You would call that "free", but Linux calls it "used".' is IMO the best way I've ever seen of explaining the issue. It also reveals the solution: call it free! It is free, after all. Nobody cares about how much free ram is cache unless they are doing something really specific, in which case they should be passing specific flags, or using a specific tool, to report it.

Unfortunately there is no chance Linux will do this, as the relevant reporting tools are too entrenched (and with good reason).

hernan7• 15 years ago

Maybe add a new (end user-friendly, preferably graphical) tool to report memory usage the user friendly way? Then vmstat can be left as is, and end users can be directed to use the new tool.

By the way, this behavior is not Linux-specific. AIX and other Unices have been doing this forever. And end users (who should have known better, BTW) would complain too. The only machine that I remember as showing a sizable percentage of the memory as "free" was a just-imaged top of the line server. Half a gig or RAM, and almost all empty! -- this was the early '90s.

Edit: Just noticed that graphical tools that report memory usage in user-friendly terms already exist (gnome system monitor). My bad, I always use vmstat and sar myself, and only recently found out about /proc/NNNNN/status

wrs• 15 years ago

Or just rename "free" to "wasted". I usually tell people the free number indicates how much RAM they paid for but aren't using.

sliverstorm• 15 years ago

It is admittedly difficult to re-learn what you know about computers, and what Windows has taught people (including myself) is that RAM usage must be below ~75% for optimal performance. It has been my experience that paging becomes much more blocking above that.

ludwigvan• 15 years ago

    free -m | awk 'NR==3 {print $4}'

ktf• 15 years ago

It looks like gnome-system-monitor reports the "correct" amount of free RAM. I'd imagine most newbies would use that rather than popping open a terminal and typing "free," right?

NonMint• 15 years ago

You'd be surprised. Some users who use Linux consider themselves to be pretty adept, and I can easily see this problem being an issue for people who perceive themselves to be clever.

JonnieCache• 15 years ago

How does this compare to darwin's memory management?

Florin_Andrei• 15 years ago

It's merely a matter of definition - whether to include the disk cache in the "used" memory or not.

radu_floricica• 15 years ago

I have had the same problem, plus some, when trying to guesstimate memory usage in vps servers. The caching is transparent, so free always reports zero swap. ps gives sensible numbers, but the total used memory is neither RSS nor VSZ summed. I've heard about library memory being double-counted, but I've found no way to accurately get the numbers. In the end I pretty much gave up and settled for the settings that worked.

tl;dr: on vps things can be even uglier.

stcredzero3• 15 years ago

Hmm, this situation is like owning an electric vehicle. The manufacturers are passionate about technology, and they invented a flywheel that can store massive amounts of energy, giving your vehicle incredible transient acceleration response. They ship many vehicles with this new feature, but unfortunately, they forgot to account for what this would do to the battery charge display after starting. As a result, their reps keep on having to explain that the vehicles are not rapidly leaking charge shortly after starting -- instead that energy is being placed in the flywheel where it can greatly increase the car's responsiveness.

Whoops!

The difference between the scenario above and Linux's disk cache? Techies blame the former on the car company but the latter on the end-users.

Andys• 15 years ago

If only electric cars were free like Linux.

stcredzero3• 15 years ago

My analogy has more to do with approaches to UX than it does with economic model.

jawee• 15 years ago

All this does is make me regret paying more for extra memory... With a dozen programs open, I am sitting at 2.3GB free, according to this. I guess it was wrong to splurge for that extra memory.

jefe78• 15 years ago

I love explaining it! The response is priceless, "Wow, that's pretty cool!" "No, that's Linux."

spudlyo• 15 years ago

You can't disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn't! Disk cache makes applications load faster and run smoother, but it NEVER EVER takes memory away from them! Therefore, there's absolutely no reason to disable it!

Never say never. You can and should disable/purge linux buffer caching in certain situations. See O_DIRECT and/or POSIX_FADV_DONTNEED. If I'm writing a database, I take care of my own caching, I do not need the OS doing it as well. Also, if I'm doing a huge rsync of a directory tree, just because I just read every byte of every file in that tree, I most certainly do not want to load all of that crap into the linux buffer cache. It will evict my already warm set, as well as potentially cause a ton of swapping.

js22• 15 years ago

Funny, I just noticed this in the Mac OS X rsync man page for the very first time:

       --cache
              Apple specific option to enable filesystem caching of rsync file
              i/o Otherwise fcntl(F_NOCACHE) is used to limit memory growth.

Nick_C• 15 years ago

There was a patch for all architectures, but it got dropped after 3.07 (current is 3.09). Andrew wrote "I don't feel this patch is general enough nor straight-forward enough to make it into rsync."

See these for more: http://insights.oetiker.ch/linux/fadvise/ https://bugzilla.samba.org/show_bug.cgi?id=7004

Tired of explaining Linux's seemingly obtuse memory usage to confused newbies?

Related Stories