Preparing for nonvolatile RAM

http://lwn.net/SubscriberLink/498283/0470778058f0ed2c/

by willvarfar • 14 years ago

76 90 14 years ago

What you really need is stick a virtual memory manager on top of NVM instead of RAM. In this case NVM works like disk and RAM as the same time. In your file system you just have a /ram file which is used as system memory.

Then, fun things start to happen: You no longer need disk buffers, since your RAM is your disk. And mmap()ing no longer consumes "RAM" because you just map a part of your file to whatever virtual address you want. You need no swap. You never have to swap in or out. You never have to sync your disks or observe buffer dirtiness (if only on CPU cache level) because your buffer is a part of your file - once you write to it the file is already updated.

Of course, software will have to adapt: prefer mmap()ing file to read()ing one; use persistent memory structures. The goal is to minimize copy-on-write by either read-only access or safe in place data transformation.

carterschonwald• 14 years ago

While I look forward to that future, realistically nvm ram is going to be at least as pricey as ssds or volatile ram, which means that a more likely in between approach that tries To minimize the amortized io between fast nvm (our ram replacement) and cheap and relatively slow nvm (hd or ssd).

guard-of-terra5• 14 years ago

If it would cost as SSDs do, it's pretty simple then: you mount NVM as /, and /var & /srv are still on disk. All your programs lose start-up time and most of swapping.

If it would cost more, you can use it as a persistent block cache (that survives reboot), but it doesn't make much economic sense since nobody is willing to pay money for just faster start-up and warm-up after reboot. If you have 8G of NVM, you could just read 8G off disk in something like 30 sec and be settled with regular RAM.

gouranga21• 14 years ago

Crazy futurist rant...

Traditional operating systems such as Linux and windows are 100% dead when non volatile memory comes along in force. Paradigm shift time.

There is pretty much no reason to use any filesystem APIs or filesystem any more. You just keep your data in the process address space - its just not going to go anywhere. Just make a data segment persistent across processes and you can survive restarts. If you backup, you can just dump the address space. Screw hard disks as well. i imagine some form of rpc will be in place between processes so they can talk to each other and that is it. Lots of small redis instances would be a similar concept.

Imagine an mp3 server process which can provide persistence in the heap, metadata and decoding services and you're there.

it'll be like a small internet inside your machine.

Lisp would fit nicely in this world. Imagine A persistent root environment. load a defun once and it's forever. Teracotta do something similar with java.

Then again I could just be insane.

guard-of-terra5• 14 years ago

Being able to reboot is nice in case you screw up really bad. Having a transparent storage (file system) instead of opaque cross-linked in-process structures is a good thing too.

gouranga21• 14 years ago

I disagree.

If you use a safe programming language which doesn't piss over memory (I.e. Haskell, Python, Ruby, Lisp etc) you won't need to reboot. Just reload the broken function into your environment and carry on. As for data, the same thing can happen to your disk...

The filesystem only exists because we couldn't put data in process due to the cost of memory. We cram everything through the filesystem API if it fits or not due to this.

It's why our machines are slow and primitive.

Consider the case of video - it is better represented as streams of audio, picture and metadata. To get this from a filesystem, we have to mux them all into a single stream and then demux them afterwards and hand them over to other APIs to process. This just wouldn't need to exist.

lubutu• 14 years ago

If all the non-volatile in-process data structures have the same semantics, with type annotations etc, one can imagine those structures being just as transparent and ready for inspection as modern filesystems are.

ajross6• 14 years ago

I'll buy the crazy part. The "100% dead" bit loses me right off the bat. Surely there will be space for hardware abstraction, process models, memory protection mechanisms, networking, etc... in your futurist OS. And, amusingly, that code is already there in the OSes of today! Storage management is merely a big subsystem.

And as for dropping "files", I think that's missing the point. Files are just pickled streams, and there will always be stream metaphors in computing. How else do you represent "inputs" to your program (which, remember, still has to run "from scratch" most of the time, if only on test sets)?

I guess I see this as a much narrower optimization. A computer built out of perfectly transparent NVRAM storage is merely one with zero-time suspend/restore and zero suspended power draw. That doesn't sound so paradigm breaking to me, though it sure would be nice to have.

gouranga21• 14 years ago

the stream metaphore isn't appropriate for all types of data. memory allows a storage paradigm to be picked for the task at hand.

Consider redis which is a great example of this. How do you store redis' data efficiently? Well it turns that aof files are slow to start up and disk backed virtual memory is slow. The problem goes away instantly with NVM - the jib is done with no filesystem api used.

Cushman4• 14 years ago

I think you're assuming a level of opacity to the OS that is true theoretically, but not realistically. Conceptually we can treat the computer as a black box which does the same thing, eventually, whether it's using cache, RAM, HDD, or the network, but realistically the limitations leak out all over the place and are embedded all over user-facing workflows in the form of opening, saving, uploading, and such things. They may always be happening in some sense, but there is no intrinsic need to involve the user in them.

Assuming that NVRAM becomes dense enough to replace storage in practice -- which is a big assumption, but it's happened to tape and is happening to hard drives right now -- concepts like launching a program, opening and closing a file, even booting will become mostly academic. Certainly they'll be of no crucial interest to users, to whom the distinction between what something is and what it does has never made that much sense.

Sure you could apply all the same abstractions over the top, but if you were designing your OS from a blank slate, why on Earth would you? And it will only be a matter of time before one of those blank-slate OSes is compellingly superior enough to the old-school paradigm, and users will start switching en masse.

crusso5• 14 years ago

> There is pretty much no reason to use any filesystem APIs or filesystem any more

Don't agree. Filesystems are used more for organizational purposes than anything. Most of my folders are logically named and organized rather than physically based because of the types they contain. Probably the filesystem API will evolve into something more tag-based than hierarchical, but you never know. The filesystem hierarchical folder metaphor is re-used in tons of places where there's really no need for it physically. Why? Because there's a logical need for it.

Additionally, file system APIs provide lots of useful abstractions such as opening resources, closing them, reading to, writing from, appending to, etc. When you go to do your backup in process space, which data is in a self-consistent state that can be copied? You'll need something like a filesystem API to coordinate.

gouranga21• 14 years ago

Tagging works on a heap. Heirarchies are terribly inefficient for categorising information. The metadata is more important than the file location. Look at the way music is stored on smartphones.

Ever heard of STM?

Splines• 14 years ago

> Traditional operating systems such as Linux and windows are 100% dead when non volatile memory comes along in force. Paradigm shift time.

I doubt they'd be dead, they just won't be able to do the new things that NVM would allow you to do (at first, anyway).

At a minimum, mobile battery life could be better, depending on how much power is usually used to keep things going. Get an event from a radio or button and the system can instantaneously wake up.

The magic trick that the NDS does when you close its lid will become universal.

batista• 14 years ago

>Traditional operating systems such as Linux and windows are 100% dead when non volatile memory comes along in force. Paradigm shift time. There is pretty much no reason to use any filesystem APIs or filesystem any more. You just keep your data in the process address space - its just not going to go anywhere.

Yes, only we want to have those data shared among many processes, with abstractions, names, rich metadata etc. (Not to mention different machines, backups, etc).

It's not like we are currently forced to use filesystems because memory is volatile.

Actually, you got it backwards, we use filesystems specifically in NON VOLATILE memory, that is hard disks.

Detrus2• 14 years ago

This could be two decades away. NVM will first be introduced with limiting amounts of storage so there will still be other hardware.

Hopefully current OSs will be re-imagined anyway just due to the vast time difference.

gouranga21• 14 years ago

I agree about it being a way off but I think more 5 years than 20. Look at the adoption rate of SSDs

sliverstorm• 14 years ago

Traditional operating systems such as Linux and windows are 100% dead when non volatile memory comes along in force.

This implies there is something rearing at the starting gate to replace them.

gouranga21• 14 years ago

There will be and I'm going to jolly well be the person writing it :)

(I'm going to try - 10 years of embedded followed by 10 years of business facing is a good foundation and I've spent the last 15 years on the problem in my mind waiting for the technology to arrive)

marshray• 14 years ago

> There is pretty much no reason to use any filesystem APIs or filesystem any more. You just keep your data in the process address space

The Windows NT kernel is primarily a filesystem-backed address space for committed RAM. Originally you actually had to have a pagefile at least as large as physical RAM. Except for nonpageable kernel structures, all the program accessible RAM was part of a memory-mapped file. [[EDIT: There's plenty of text from Microsoft that implies this, (e.g. "you should set the size of the paging file to the same size as your system RAM plus 12 MB...because Windows NT requires "backing storage" for everything it keeps in RAM") however, offline discussions have convinced me that it was never strictly true.]]

This was back when drive capacity was "more larger" than RAM capacity and disk bandwith was "less slower". The kernel has evolved away from this design a bit, but it did bring a certain purity. For example the filesytem cache and virtual memory paging system could be largely the same thing.

> You just keep your data in the process address space - its just not going to go anywhere. Just make a data segment persistent across processes and you can survive restarts.

This more or less what happens when the kernel bluescreens and the page file was at least as large as RAM. It makes debugging kernel crashes easier. (Spare me the Windows jokes please I'm not advocating for it, just saying this part of it that no one ever sees had a relatively elegant design opposite to what crazy futurist rant suggests.)

jws• 14 years ago

One might think about the paradigm shift when hard disks became so cheap that tape drives no longer made sense.

• Tape drives still make sense, to some people that store insanely large data sets with extremely infrequent (or never) access.

• I do not mourn my tape based backups for a single instant when using my rsync based backups. Life is good.

• Despite tape being dead for most of us, we all know a program called "tar". We still make tape archives, just on other storage.

The paradigm survives on virtual tapes because it is a useful cognitive model. Sure I could make a block file be a virtual disk and put a filesystem in it and send you the files that way, but you'd rather have a good old "tape" archive.

Likewise, disk filesystems are not going to go away if disks go away. They are too useful for reasoning about problems.

Scratch the surface of an iPad. It is full of files, yet empty of disks. Go to Linux, land of speciation, try to find a persistent storage system for flash memory which does not treat it as a disk. You will find some filesystems optimized for flash, but you will come up (nearly) lacking for a completely new way of looking at storage.

Persistent full speed RAM should be enough of a change to spawn that new thing, but I'll bet people keep the "real" copy in a filesystem for a long time. When that alpha particle corrupts a bit and trashes your clever RAM based data structure, what are you going to do? I'll reload from my file.

aiscott• 14 years ago

Personally I don't like the idea of "image" based computing very much. I mean, it's fine in a lot of circumstances, but the file system provides a nice broadly compatible database that is consistent across programs.

I think the need for such a well defined and accepted user organizable data store isn't going away regardless of the underlying storage medium.

I like that my file system can be reasoned about in very concrete ways. I'm okay with using a tag based system like gmail, but I find it very flat and more difficult to organize vs traditional hierarchy of folders.

And btw, GET OFF MY LAWN!

jpadkins3• 14 years ago

in this world, how do you programs on different computers reference the same piece of data? ipv6 address/memory address?

the whole path/file thing is really useful for distributed computing.

Also what would users use for object naming(especially across address spaces)? How do I search for a presentation or spreadsheet without a reference to it?

gouranga21• 14 years ago

a) global address space.

b) no it's not. Distributed computing is normally message based.

c) you use an index in (a) and pull it into your local scope.

snprbob865• 14 years ago

This seems like one of those "sounds good on paper" ideas, like shells that pipe objects. In practice, file and process oriented systems are separate from REPL systems on purpose.

The most important property of a filesystem or the Unix-style pipes&filters that permeate them is the fact that serialization is a fundamental property component of persistence and communication. Any finite quantity of space can be addressed by a linear scheme and any unbounded sequence is inherently linear. Consider what happens as soon as you want to send a stream of data across the wire. Or checkpoint an operation. You need to store stuff in a non-volatile and linear form. And since data outlives code, you better invest time into thinking about that linear form.

Now, if you use immutable data structures (ie. non-cyclical) with a proper linearization and corresponding reader/printer (like in Clojure), you can get many of the same benefits, but you still need to have a whole bunch of other things to worry about. Just look at the weird things that Clojure needs to do with print-dup and the like.

I'm not saying that there isn't a kernel of a good idea in here. I'm just saying that it's going to have a lot more in common with filesystems and traditional shells than you'd first expect coming from a Lisp REPL school of thought.

gouranga21• 14 years ago

> shells that pipe objects

Powershell rocks!

No wait, sorry it doesn't. I agree!

abecedarius• 14 years ago

http://www.cis.upenn.edu/~KeyKOS/ looked pretty much like that. The implementation treated RAM as a cache of the disk, IIRC. This seems like a better way to live than Unix, to me, but you can't say it hasn't been tried.

gouranga21• 14 years ago

That's exactly the sort of thing I'm thinking.

bigiain• 14 years ago

Hmmm, I'm remembering back to the Apple Newton's "soup" memory… (You don't want each process to have exclusive access to it's own data, sometimes you want different processes to access the same data)

rwl4• 14 years ago

A descendent of Smalltalk seems like it might fit the bill here..

ricardobeat• 14 years ago

Is it possible that the switch to no filesystem in iOS userspace (and probably OSX in the future) is in anticipation to this in some degree?

<tinfoil hat on>

gaius• 14 years ago

Crazy futurist rant...

Haha, this is exactly how the IBM AS/400 works!

http://en.wikipedia.org/wiki/Single_level_store

gouranga21• 14 years ago

thanks for the pointer.

I now to you IBM (but not the COBOL bit)

raverbashing2• 14 years ago

I will only believe this once it's sitting in my desk

And, for now, it won't change filesystems much. Unless you can get a similar ammount of it as a disk (or maybe a compromise, let's say today around 8GB ram is common, and 1TB of HD, then if you can get around 128GB NVM this can be your new 'SSD')

It is of course, a very important development, and may make things faster

sp3322• 14 years ago

How about 240 GB almost as fast as DDR 200 (PC-1600) RAM? http://www.newegg.com/Product/Product.aspx?Item=N82E16820227...

Edit: according to reviews it uses compression to boost its bandwidth (but not capacity). Still seems like a decent tradeoff.

raverbashing2• 14 years ago

Wow, very tempting

Too bad it doesn't fit my MBP =)

egillie• 14 years ago

Crazy! I was looking into how much it would cost to get 500GB of RAM, looked like it was going to be around $3500. I hadn't even considered this...

notatoad• 14 years ago

You can easily run a desktop pc wih a full featured distro like Ubuntu desktop off a 16GB disk. You certainly don't need 128GB.

leoc• 14 years ago

Obigatory mention that the DG Nova http://en.wikipedia.org/wiki/Data_General_Nova was the star of http://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine The Soul of a New Machine.

vicaya• 14 years ago

Commodity NVM is huge for big data transactions.

  Bye bye WAL
  Bye bye fsync
  Hello NVM replication
  I think I'm gonna cry

sweis3• 14 years ago

Widespread adoption of NVRAM may require a significant change in security models, since data once assumed ephemeral may be persistent. For example, it may be trivial to recover cryptographic keys from a running system.

(Disclaimer: I'm working on a solution to this.)

kijin• 14 years ago

This. I'm surprised nobody else mentioned security. Instead of chilling DRAM sticks immediately after shutdown and trying to read its contents with specialized tools, people could just take the NVRAM out and analyze it at leisure.

I don't think it will be too difficult for security software to wipe their keys from memory before shutting down, and many programs already do this. But so much more would remain vulnerable unless the decrypted data structures were also wiped from memory. Implementing effective security with NVRAM-equipped computers might therefore negate much of the benefit of using NVRAM in the first place.

aidenn0• 14 years ago

There are already solutions out there for the paranoid who are afraid of cryogenic attacks on DRAM

sweis3• 14 years ago

Yes and no. There are some solutions out there, but they are typically not ready for production, require custom hardware, or make assumptions about physical security controls.

(Disclaimer: I am biased since I am working on this.)

gouranga21• 14 years ago

That's fairly easy to deal with - TPM module in PCs does this already.

sweis3• 14 years ago

The TPM does not provide full memory protection. TPMs are primarily used to attest the state of running software. It does not prevent someone from reading memory contents.

mmphosis• 14 years ago

What about processors that rather than accessing external RAM and levels of cache, instead a large amount of (register) memory (nonvolatile or not) is directly included within each CPU?

oh_sigh• 14 years ago

Can anyone tell me why we don't just hook normal RAM up to a small rechargeable battery such that it can maintain its state during a power loss. Alongside that, there is an equivalent amount of flash memory. The flash is never used, except when you get within, let's say, 5% battery life, at which point the entire contents of the RAM are dumped to the flash. Then, on system start up, if the RAM is still loaded, swell. If there is a ram image on the flash drive, load that and continue as normal.

Isn't this essentially NVRAM? What are the downsides to it?

shabble• 14 years ago

We do, something like http://techreport.com/articles.x/16255 or http://www.anandtech.com/show/1697/5 although I don't think they actually have a full NV backing store. They definitely exist though, and have done even before SSDs, it just had a normal disk hanging off it.

Battery-backed cache has probably been around even longer in write caches for large RAID systems.

The 5% power and shutdown is also how the Macbook laptops handle sleep. Sleep mode is a low power nothing-but-ram mode, and when the battery gets too low, it goes into 'safe sleep', basically a dump-to-disk all-powered-off hibernate.

The main reason it's not all that common is that for the sort of workloads you're prepared to pay for a shitload of RAM, you're probably just using it as a cache for a DB or some monster app, and actually keeping it around isn't that much of a priority. You've got failover somewhere else in the stack, and it's one less thing to buy and maintain.

The other critical flaw is that there is a (potentially huge) performance hit in presenting as a disk vs hanging off the northbridge MMU. Even the latest in new fancy SATA is hilariously slow compared to the actual memory bus (6Gbps for SATA3 vs maybe 100Gbps for DDR3[1]), and having all the filesystem abstraction on top, as in the titular article of this thread mentions, is a whole lot more overhead.

So yeah. We can. Sometimes people do. But it's probably easier and better to just stick it in the actual RAM slots, and use it differently for everything except 5-second boot times.

[1] https://en.wikipedia.org/wiki/List_of_device_bandwidths#Stor...

toddh• 14 years ago

If you want battery backed ram you might as well use a UPS. There are also corruption issues. A failure can leave data structures in an inconsistent state that applications typically don't worry about in RAM.

Preparing for nonvolatile RAM

Related Stories