Linux ate my RAM

http://www.linuxatemyram.com/

by wallzz • 13 years ago

194 125 13 years ago

linuxatemyram.com

jaseemabid• 13 years ago

Lovely! This domain could iframe that SO question :)

daGrevis2• 13 years ago

What question?

Schiphol• 13 years ago

Why, that one, of course.

daGrevis2• 13 years ago

The page seems to be down.

Here's how it looked: http://web.archive.org/web/20131011053519/http://www.linuxat...

makomk2• 13 years ago

This is actually wrong. For some reason (probably because of the way it's implemented internally) Linux includes shared memory allocations in the "cached" figure. If you calculate the amount of free memory using "free -m" in the way this web page advises, you'll think that memory is free when it really isn't - unlike true cache, shared memory cannot be discarded when the space is needed for something else.

I've actually had buggy desktop applications use several gigabytes of RAM as shared memory, so this can make a big difference.

_bz2r• 13 years ago

For some reason (probably because of the way it's implemented internally)

(this is tautologically true)

cma• 13 years ago

he means the rationale behind developing it that way was due to the way other stuff is laid out internally, not "the number shows up because the existing code codes for it and causes it to show up".

gue5t• 13 years ago

Linux also counts files on tmpfs as part of the cache number--it's really annoying that no number in top can serve to show you a decent approximation of how many page-size malloc()s you can do before OOM (with overhead and fragmentation and so on, of course it's infeasible to get an exact number).

marcosdumay2• 13 years ago

It's worse. No line of /proc/meminfo has that value, and since not all of the cache can be instantly discarded (I didn't know that), I don't see any combination of them that would help.

That sentence:

There are no downsides, except for confusing newbies.

Just isn't true. There is the downside that we can't know how much memory is available.

tonylampada• 13 years ago

So... what exactly do I have to type on my command line to see how much RAM can actually be used by applications?

mistercow• 13 years ago

I used to have a similar problem on OS X. When I was working on a 32-bit app that could eat tons of memory, I ended up using a bunch of weird heuristics and allocation tracking just to get a probable assurance that the program could recover properly from running out of address space.

jbogp• 13 years ago

Also HN ate your server apparently.

Morgawr• 13 years ago

Best way to check how much ram is actually free is to use a higher level tool like a task manager (from various DEs) or a command line tool like htop, they report the actual memory usage of processes

v-yadli2• 13 years ago

https://www.kernel.org/doc/Documentation/sysctl/vm.txt

You can drop cache by sending a byte into drop_cache.

mprovost• 13 years ago

But why would you do that? (The only time I've ever used that is when benchmarking and you need a clean cache between runs). Otherwise the kernel will just empty it as needed when you run a program.

jrkatz• 13 years ago

I can't answer for everyone, but I can tell you why I have a cron job dropping my cache every several minutes:

The application I develop consumes most of the ram on my machine when running, and takes several minutes to rebuild and start up to begin with. When most of my memory is being used for the cache, this process takes several minutes longer, because I'm making millions upon millions of calls for more memory -- and each one has to get some of that cached memory back for itself. If I simply drop the cache all at once, every minute, it takes a split second. If I shrink it over a million increments, it takes around a minute.

Even typing that I feel I must be doing something wrong; and yet, it worked.

v-yadli2• 13 years ago

First, it's not why. It's that this is possible. Please don't say "this is impossible because this doesn't make sense". Second, I agree with you. Normally people don't need to do this unless they hate their hard drives. :-) But there's some special cases, for example when the disk is "volatile", or when you want a huge chunk of continuous physical address and you want to tell the OS to stay away and don't touch any pages inside.

PS: An interesting thing is that Windows people always seek for a way to enlarge disk cache. It's fine on server versions, but on desktop versions such as Windows 7, if I don't remember wrong, disk cache is capped at ~4GB(not sure if globally or per file). Oh well.

Damogran6• 13 years ago

FWIW, it's relevant to OS X too (even Mavericks)...'sudo purge' will reclaim some of the resources.

sounds• 13 years ago

OS X does not behave exactly in this manner, but is indeed confusing.

Linux never needs a 'purge' or equivalent. Doing a 'drop_cache' only slows the system slightly because nothing is cached. 'sudo purge' can prevent OS X from using swap – i.e. running out of RAM even though there is RAM it could reclaim. This behavior has been tweaked but not fixed in Mavericks.

Zr40• 13 years ago

Actually, purge is completely unrelated to swap. Purge forces disk cache to be flushed to disk, and can be used to approximate a cold disk buffer cache scenario for performance analysis.

Some small amount of swap usage (compared to total RAM) is perfectly normal, by the way. More cache, for instance, is a better use of RAM than avoiding swap usage by keeping never-accessed pages present.

free652• 13 years ago

Windows loves to cache and "prefetch" apps, I still need to figure out how to disable prefetching since I have a very fast SSD.

csmuk• 13 years ago

Why? All it does is make it go faster.

TBastiani• 13 years ago

And drain your battery for no reason.

sp3322• 13 years ago

Instructions for Win7. http://www.tomshardware.com/reviews/ssd-performance-tweak,29... Why not leave it on, though? RAM is still faster than SSD.

stephencanon3• 13 years ago

Memory is still an order of magnitude faster than your “very fast SSD”.

alcuadrado• 13 years ago

just one?

dingaling• 13 years ago

And memory is still an order of magnitude more expensive than SSD, per GB.

I just bought 4 GB of PC-6400 for a laptop at £40. A fast 128GB SSD is £100.

Summary: use RAM judiciously

rnnr• 13 years ago

Windows7 are SSD aware. You dont have to do anything about it.

jheriko2• 13 years ago

right. so there is a bug in linux. gotcha.

annnnd• 13 years ago

No. Well, not technically... There is a UX bug in Linux. Not the only one at that, but they don't count much in Linux-land.

hansbogert2• 13 years ago

Just a difference in semantics -- I mean, c'mon UX in a utility like top? which is probably specified in some Unix convention anyway. Pure BSD Unix do this as well in their own way.

CmonDev• 13 years ago

Free is used, black is white, truth is a lie, lie is a truth.

gmuslera• 13 years ago

Leaving your memory with no use at all would be the bug. It is a free resource that could be used to improve system performance at basically no cost (the disk is already read, the memory is still available immediately for any app that request it, etc), so it is a good default to let linux do it.

But could be cases where it could be side effects of it (i.e. running in a VM could make less memory available for the host, and it could be caching disk already) so you could consider to disable it sometimes.

jheriko2• 13 years ago

it confuses users. its a bug. thank you for the -4 points... :)

caw• 13 years ago

Wow, I can't believe how many times I've had to explain this to people as they log on to shared VNC servers. "I thought this had 16G of RAM free, I asked for a 16G machine!". "It does, stop trying to read top!"

Now I'll just point them to this website.

pearjuice• 13 years ago

Then you should http://downloadmoreram.com

blowski• 13 years ago

Haha, that's brilliant. From their Twitter account:

Have an iPhone 3? Download RAM twice to make it a iPhone 5.

https://twitter.com/DownloadMoreRam/status/38502626000858726...

10098• 13 years ago

Crap, I wanted to post that!

puppypilgrim2• 13 years ago

Brilliant. I love it.

unimpressive• 13 years ago

When my circle of friends found that site it became an in-joke. I'd never heard of it, and eventually they came up to me and asked "Can you download more RAM?".

I don't think they were expecting my answer:

"Sure you can, just use plan 9."

nraynaud• 13 years ago

that's funny, like avery complex or "smart" algorithm, if it works it's cool, but when something doesn't work, it makes it very hard to pinpoint a culprit.

I'm more and more inclined to either use stupider algorithm that at least I can explain to the user, incremental stuff where they can check intermediate results or taking into account the fact that this algorithm has to be conveyed to users in one way or another since inception.

I guess in this case a measure of the swap activity would be a good indicator that the current limiting factor is the RAM, but it also has to be credible. Somehow having 16G written on the box of the ram and 7G written in the "available ram" of a software makes everything more credible that saying "I have a pressure of 1.5madeupunit on the ram front". Human factors, again.

lamontcg• 13 years ago

I've been doing performance analysis and debugging as part of being a unix/linux sysadmin now for something like 20 years. And even though I have many well-cherished anecdotes about explaining the VM free list to software architects who were the go-to guys for the vision of our entire global platform, complete with PhDs, I still don't want to go back to the bad old days without a shared buffer cache. There's a reason why that's there, and you don't want to go back to 1992-era memory management.

It also makes perfect sense from the perspective of writing the software. The free list really is the list of VM pages that have been free'd and that has a specific meaning, and changing that name internally would be terrible. You just need to understand how the machine actually works.

andmarios4• 13 years ago

This is the worst piece of Linux advice ever. I use Linux for many years and every time I had a memory leak or a process using too much memory, every google search about the problem would lead to some "guru" giving this advice.

This explanation which has been replicated thousands of times on every linux forum on the planet, prevents people from solving real problems. It started as a useful piece of information but nowadays it is spam.

jlgreco4• 13 years ago

How does this understanding prevent people from solving real problems, simply by virtue of it being commonly cited? I don't understand your complaint.

marcosdumay2• 13 years ago

I understand this is a complaint about Google. It won't help you find pages that cite a problem when a diffrent, but similar problem is too popular.

I've seen that happening several times, but I'm not the GP, so I may still be missunderstanding him.

alandarev• 13 years ago

Exactly that!

I had similar experience with MacOS memory issues, and all the answers I was getting: "MacOS is superior to Windows, its just Cache, the memory is actually free". Yet my applications could not allocate memory and were hogging whole OS as a consequence.

Back on topic, never had similar 'cache' problems on Linux though.

hippich2• 13 years ago

That's how I check true free memory available:

  root@joe-pc:~# free -m
               total       used       free     shared      buffers     cached
  Mem:         32057      30347       1710          0         1312      21568
  -/+ buffers/cache:       7466      24591
  Swap:            0          0          0
  root@joe-pc:~# sync; echo 3 > /proc/sys/vm/drop_caches
  root@joe-pc:~# free -m
               total       used       free     shared      buffers     cached
  Mem:         32057       5628      26429          0            4        891
  -/+ buffers/cache:       4732      27325
  Swap:            0          0          0
  root@joe-pc:~#

dice• 13 years ago

Yikes! Hopefully not on production systems. I've had fileservers which needed 60+GB of dentry cache to serve up files in a reasonable time, something like that could bring the site to a crawl for 10 minutes or more.

hippich2• 13 years ago

Good point. I am doing it on my laptop and server without load. So this is not a big problem in my cases.

bediger40002• 13 years ago

How bizarre. In the 1993-94 time frame, one of the big talking points about Windows NT was that it was the only OS with an integrated file system cache. That wasn't true even at that time, as Solaris also had such a cache, but such is the nature of propaganda.

I'm tickled that what once was a big "technical" advantage becomes a major source of confusion. Presumably, OSes of Windows NT heritage have retained file system caching, which offers a huge performance benefit. Why don't Windows users have confusion over this issue? Doesn't Windows offer an easy way to check or is the whole thing presented clearly, or is Windows so complicated not many people get this far?

I'll also draw a parallel to DUI - without a breath-o-meter, DUI is a lot like driving while distracted. It's impossible to quantify, and therefore not that much of a crime. With a breath-o-meter, or with "top", it's possible to put a number on drunkeness or lack of free memory, and the whole thing becomes a problem.

chiph• 13 years ago

From what I can tell, Windows does not show the disk cache allocation in Task Manager. So even though all the RAM is likely in use, that is hidden from the user. You can probably see it in PerfMon, as that is less "typical user" oriented.

rplnt• 13 years ago

It's not in the graph but it's there in the "Cached" field.

tracker1• 13 years ago

In windows 7, take "available" and subtract "free", that's your cache usage.

Windows Vista started using RAM very aggressively for caching, including pre-filling cache with frequently used pages (apps/data) from disk... ReadyBoost was basically an extension of this...

Lagged2Death• 13 years ago

Why don't Windows users have confusion over this issue?

Windows' Task Manager has generally counted memory occupied by the disk cache as "available."

tanzam752• 13 years ago

Windows' Task Manager has generally counted memory occupied by the disk cache as "available."

Windows Vista got lots of complaints about memory usage, because Task Manager listed memory as Total, Cached, and Free. People saw that "Free" was close to 0, and complained that Windows Vista ate all their RAM.

Windows 7 changed this to: Total, Cached, Available, and Free. People saw that they had a lot of "Available" memory, so they didn't panic. Well, most of them didn't panic.

Windows 8 removed the "Free" line entirely, and now lists memory as: Total, Available, and Cached. If you care about the actual amount of free memory, you can run Resource Monitor or Process Explorer.

bjourne• 13 years ago

Because there are a lot of things in linux/unix-land that is completely braindamaged. But they have been with us for so long that they become impossible to change. Like dotfiles, the ps command and Linux memory handling.

Windows 8 has a nice graph which shows memory usage over time in which the filesystem cache is not visible. If it was included, memory usage would constantly hover at about 90% which wouldn't be very useful and very confusing.

When Windows XP was introduced famous tech journalist John Dvorak complained that "System Idle Process" was using 99% of his cpu and slowing down his computer. In later Windowses, System Idle Process is no more.

Also even when you subtract the fs cache and get number of megabytes free memory you have available. That number has almost no bearing on how many number of megabytes an application of yours can allocate. Linux memory management is much more complicated than that.

ris2• 13 years ago

Linux memory management is much more complicated than that.

I don't know if I'm being trolled, but memory management in any modern OS that makes proper use of shared memory is fundamentally very complicated. I suspect what is happening is Windows is lying to you to give you a simple (though essentially wrong) answer.

bediger40002• 13 years ago

Wait, so it's good to not show people everything?

Iknow my taste in user interfaces is very different than most, but this really explains a lot.

MichaelGG2• 13 years ago

Windows 7 still has "System Idle Process" showing CPU usage.

outworlder• 13 years ago

Once upon a time John Dvorak's word was taken seriously.

Whatever happened to him?

rlidwka• 13 years ago

what's wrong with dotfiles?

stonemetal2• 13 years ago

Why don't Windows users have confusion over this issue?

Because windows users don't care about how much memory a particular application is using. New Windows developers get confused by it all the time. There are plenty of questions on stack overflow where someone is using task manager to try and measure their application's working set.

gtirloni• 13 years ago

An overgeneralized question deserves an overgeneralized answer. Well done.

nly• 13 years ago

Forget the kernel, someone really needs to do chromeatemyram.com

FlyingAvatar• 13 years ago

I hate the invention of "burstable" RAM on VPSes for this reason. Eventually your IO cache grows until it has most of your guaranteed RAM. Now every new process is playing a game of Russian roulette with your hypervisor to see if it can actually allocate into your "burstable" RAM what the kernel is reporting is freely available.

vajrabum• 13 years ago

Actually this is a general issue when running multiple Linux VMs under a hypervisor. In that case you don't want them filling up RAM with prefetch data from disk because that's likely to cause contention and reduce page sharing between VMs.

rdtsc3• 13 years ago

A quick practical answer is to use htop and look at the memory bar.

* Green = resident memory used (probably what you want most of the time)

* Blue = buffers

* Yellow = cache

ChuckMcM• 13 years ago

+1 for htop, it not only shows excellent memory stats in an easy to absorb way but it also shows CPU cores usage. Definitely one of my "wtf is going on?" tools.

hisham_hm• 13 years ago

htop author here. It was nice seeing the table in the article with the memory split in three colors and thinking "well, but that's how I already split it in htop, isn't it?".

The "Linux calls it" column should probably be called "top calls it"...

rdtsc3• 13 years ago

Thank you for writing htop, it is very useful tool.

I just about fell off my chair when I realized I could click with my mouse on the columns to sort them!

sciurus• 13 years ago

I'll give a pitch for atop as well.

Here's a screenshot of it showing systemwide and per-process memory information: http://www.atoptool.nl/images/screenshots/memory.png

Here's a case study of using it to observe a memory leak: http://www.atoptool.nl/download/case_leakage.pdf

usamec• 13 years ago

Ehm. But sometimes stupid linux kernel prefers moving something usefull to swap and caching some shit instead of keeping stuff in ram and not doing caching.

So linux sometimes eats your RAM (but in different way).

Amadou2• 13 years ago

For situations where you have a time-critical app, you can pin the app's memory into RAM with the mlock() family of functions.

There may be a command-line utility that lets you pin an arbitrary running a process too, I haven't looked for it on linux.

jared314• 13 years ago

Previous Discussions:

https://news.ycombinator.com/item?id=3698652 (1 year ago)

https://news.ycombinator.com/item?id=6368562 (2 months ago)

https://news.ycombinator.com/item?id=1831322 (3 years ago, no comments)

jvdh• 13 years ago

So basically this is the inverse of swap. Using memory to make disk reads/writes faster, instead of using disk to make more memory.

If Linux is consistently doing this, why would you still need a swap partition?

ableal2• 13 years ago

why would you still need a swap partition?

You don't, but if something decides to grab memory, the OOM-killer will kill processes abruptly. Swap may allow the system to limp along until there's some solution.

It's not like 10 or 20GB HD these days matter much either.

perlgeek• 13 years ago

Though of course some people prefer the OOM-killer to kick in fast instead of the whole system being unresponsive for about an hour or so.

That's why my desktop and laptop systems don't have swap enabled.

rosser• 13 years ago

Would you rather consume physical pages of RAM by storing less recently used programs and data, or by caching more recently used disk pages?

Myself, I'd generally prefer to evict the older stuff to swap in order to cache the newer stuff. But maybe that's just me.

Tuna-Fish• 13 years ago

Because you can (and usually do) have areas of memory and disk that are more or less active, meaning it can make sense to evict some ram pages to disk while still caching other disk pages.

njharman• 13 years ago

why would you still need a swap partition?

Because sometimes you (read applications) need to use all/most of your memory. Ability to swap out all the not being used memory to disk maximizes amount of memory available to applications.

lmm2• 13 years ago

You don't. Get rid of it. I've been running without swap for five years now. (Note that suspend-to-disk requires a swap partition though)

AsymetricCom• 13 years ago

ITT tech startups learn about Lunix.

talles• 13 years ago

Or you can just http://downloadmoreram.com/

(joke)

fit2rule• 13 years ago

Another thing, you know how sometimes a USB stick needs some time to un-mount? This non-sync'ed data (say, a copy of a big file) eventually has to be written to the physical disk, and through the little pipe, takes time. Find out how much of this "dirty" buffer you have yet to physically write to the device, while giving the kernel 'hints' to sync the dirty blocks quickly, with:

    while true; do sync && grep Dirty /proc/meminfo; done

Lots of gold to be mined from /proc/meminfo .. bonus points for wiring it up to gnuplot so you can get a real picture of the dynamics of the Linux kernel .. ! :)

voltagex_• 13 years ago

This seems a bit magic, what does it do? (stuck in Windows-land at the moment)

lcrs• 13 years ago

I do this a lot, on machines which are annoyingly vendor-tweaked to allow a lot of dirty pages. Copying files to USB sticks happens instantly, but the unmount then takes an age - like, 20 minutes. This is a way to get a measure of the progress of the unmount. I generally have a sleep in there as well though ;)

Sami_Lehtinen• 13 years ago

I think it's still disinformative. Disk cache also uses RAM which is allocted and reserved by applications, of course data is swapped out in this situation. o you'll need more swap, but it isn't a bad thing at all. Even OS/2 did this a long time ago, nothing new at all.

xacaxulu• 13 years ago

I just love the website design :-) It makes me happy.

richo• 13 years ago

Not strictly accurate, in the "Why would you want to do this" regard.

Ages ago I was hacking linux onto an ARM soc that claimed to have memory it didn't (it wasn't aware of the display's DMA segment). It was near the top of the address space, so generally speaking it didn't cause too many problems, but until I could debug it I needed to keep memory usage low, wchih meant disabling the disk cache. What a shitshow that was :/

dmourati• 13 years ago

I always run Ganglia on my servers. You can see memory used, memory shared, memory cached, memory buffered, and memory swapped all over time as well as total in-core memory. This has saved me (and my team) from ever having to misinterpret command line output. Takeaway, Linux is smart about using memory, probably smarter than you. http://ganglia.sourceforge.net/

Linux ate my RAM

Related Stories