Vramfs – GPU VRAM based file system for Linux

https://github.com/Overv/vramfs

by pabs3 • 3 years ago

243 77 3 years ago

github.com

riobard2• 3 years ago

would be interesting to combine this with `zram` to move compressed swap to vram instead

drpixie• 3 years ago

and you have all that GPU processing power for compressing & decompressing :)

rbanffy4• 3 years ago

But then you'd need to transfer uncompressed data over the PCIe bus.

codeflo2• 3 years ago

I assume you're tongue in cheek, but just to make a serious point, it should never make sense to swap from cheap RAM to more expensive RAM. The whole point of swap is to move rarely used stuff into storage with a lower cost per GB. Otherwise, you would just buy more RAM in the first place.

userbinator4• 3 years ago

VRAM will still be faster than HDD.

dspillett• 3 years ago

you would just buy more RAM in the first place

Unless you have hit the limit on how much "normal" RAM the current machine(s) can have/address, and/or you need extra memory for something now, etc.

Or, of course, you are just playing with something cool but ultimately pointless!

citrin_ru• 3 years ago

Unless you have a notebook/desktop with a powerful GPU and there were no option with all things staying the same but a cheaper GPU so you ended up which under-utilised GPU.

unphased• 3 years ago

It's relatively fast (probably could be faster than NVMe if optimized, i did not rtfm) and it is available. it's not the most horrible of ideas. I am fairly fairly certain lots of applications will load assets into vram and then evict them from system memory. Why not leverage this for general purpose use if it makes sense (after taking into account the security implications of course)?

ZiiS2• 3 years ago

If you pay for GPU memory for the 10% of tasks you need it; it is cheaper then system RAM the rest of the time.

riobard2• 3 years ago

You need to factor in the compression ratio to see the benefits of swapping from RAM to RAM :P

peter_d_sherman• 3 years ago

Agreed!

It would be interesting to implement this and a highly parallelizable GPU background data compression algorithm...

It could be zram (zram would be a good choice!), it could be something else, it all depends on what kind of performance/compression is desired, and how parallelizable the compression algorithm is...

KRAKRISMOTT• 3 years ago

Intel Optane anyone?

LargoLasskhyfv4• 3 years ago

I did that long ago with an IBM T60p, which had a 32bit Centrino, and only 3.25 or 3.5GB usable RAM out of 4GB. While the ATI FireGL had 256MB. At the time there was no ZRAM, but its predecessor compcache. IIRC I only needed about 2MB VRAM for the stuff I did, any capability the FireGL had wasn't supported by MESA/XORG/DRI at the times, apart from useless GLXGEARS.

What can I say? After much fiddling it worked stable, resulting in running into the limits of that POS much later. Extending its usability. Not that much, but noticeable.

Even Google Earth ran. Probably because it did anything in software anyways, on that combo of supported and exposed capabilities of the GPU at that time.

userbinator4• 3 years ago

The idea of using VRAM as general-purpose storage has been around for over a decade, although it usually makes more sense to put swap on it:

https://forums.gentoo.org/viewtopic-t-395234-start-0.html

https://www.pmenier.net/dotclear/docext/mtd/TIP_Use_memory_o...

jwilk2• 3 years ago

Over two decades, as you pointed out in https://news.ycombinator.com/item?id=11559869.

userbinator4• 3 years ago

...and that was almost 7 years ago too, during which my second link there also died; but that was before the rise of all the AI stuff and its accompanying demand for GPUs with otherwise insane amounts of VRAM.

gruturo• 3 years ago

Make that three decades ago. In 1992 I was using a utility called VGADISK to create a 176K ramdisk in the VGA memory - this was on a 286 "laptop" (rather a luggable) with 1MB of RAM, and allowed me to fit entire programs in there as long as they didn't use VGA graphics (which would instantly corrupt the content of the ramdrive).

Still online thanks to http://www.retroarchive.org/garbo/pc/memutil/index.html - a whopping 8KB zip archive, so about 30 seconds on my 2400bps modem (assuming V42 or MNP4)

mobilio• 3 years ago

Or this one

https://wiki.archlinux.org/title/Swap_on_video_RAM

Rediscover2• 3 years ago

OMG! I run in either text mode or very minimal graphics mode. This is a very appreciated comment. Now I am going to reconfigure my home machines to steal back that otherwise unused VRAM for swap.

Thank You!!

slug• 3 years ago

Is this safe ? Had the impression that undetected bit-flips were more common with GPU RAM than with CPU RAM banks, unless you have professional GPU cards with ECC.

chronogram• 3 years ago

Why would you add a dedicated graphics card to computers that run in text mode or a very minimal graphics mode? Seems expensive.

LargoLasskhyfv4• 3 years ago

Just SWAP is inflexible, especially with the way Linux usually handles buffer/pagecache, memory allocation,and countless other things, if not explicitly configured otherwise. You'd probably get the most out of it if you'd use https://en.wikipedia.org/wiki/Zram setup to use that VRAM instead.

Except if it has a very weak CPU which struggles with the load caused by compression, or power constraints, because mobile? But even then the compression is configurable.

I once used something similar on an old Thinkpad 60p with an ATI FireGL which had some 32bit Intel Centrino, and had only 3.25GB usable RAM from the 4GB. The FireGL had 256MB of which I only needed 2, and let it have 6, just to be on the safe side. The other 250MB I gave to compcache pointed to the device created by some (experimental?) blockdriver I can't remember anymore.

This wasn't automatic, didn't work on the first tries, and produced crashes. But was worth it once I had it right(¹), because that feeble thing ran much later into its limits. Noticeably. Not by some meaningless benchmarks.

(¹) much fiddling with many hexadecimal memory addresses and ranges, exclusions of the same in other configuration files, scattered over /etc/, mostly Xorg, but also other Drivers/PCI-Address spaces, to not have one BYTE of overlap, but also not wasting any ;-)

Edit: Skimming the two links phram rings a bell. It also ran Google Earth! Didn't really matter, on that GPU it probably did everything in software, and not something which the FireGL could deliver, or at least not at the times, with what MESA, XORG, DRI could do then with that card.

actionfromafar• 3 years ago

I used I think 32 megabytes VRAM as swap way back when. I think it was on single channel AGP or PCI bus, so it wasn't fast, but much faster than disk, so pretty good.

FooBarWidget• 3 years ago

In the era of huge ML models, wouldn't it make more sense to do the opposite, i.e. use main RAM or SSD as swap voor VRAM? So that we can run e.g. ChatGPT on consumer hardware.

rbanffy4• 3 years ago

A lot of low-end GPUs do that and, when you don't have enough memory to fit the working dataset in the GPU's very fast RAM, the CPU will end up swapping in and out of the GPU via the PCIe bus.

brucethemoose2• 3 years ago

This is easier said than done, though certain frameworks (like Microsoft's Deepspeed) try to do this "automatically"

dahart• 3 years ago

I don’t know about Linux these days, but Windows WDDM likes to have host-backed storage for VRAM allocations... which makes me wonder if using VRAM for swap could ever get you into a weird recursive state with allocations or swapping. It’s already the case that allocating VRAM on Windows can require waiting on swapping some VRAM to disk. What if that swap had to then go back to VRAM?

codeflo2• 3 years ago

I don't know if it's esoteric filesystem week on HN, but I love it.

[1] Yesterday: TabFS – a browser extension that mounts the browser tabs as a filesystem, https://news.ycombinator.com/item?id=34847611

[2] The day before: Linux's SystemV Filesystem Support Being Orphaned, https://news.ycombinator.com/item?id=34818040

OscarCunningham• 3 years ago

It's older than this week, but you might also enjoy 'Harder Drive': https://youtu.be/JcJSW7Rprio.

loeg• 3 years ago

Tom7 is a national treasure.

Svenstaro• 3 years ago

I once posted memefs: https://news.ycombinator.com/item?id=17799547

I didn't get picked up very much, though.

pyinstallwoes• 3 years ago

That’s really cool

wpietri• 3 years ago

Holy moly! TabFS is amazing and I missed it. Closing a tab by rming the dir is especially great. Thanks for mentioning it!

voxadam2• 3 years ago

It really does seem like a natural extension of the Unix maxim of "everything is a file"; I can imagine a modern browser embedded in a Plan 9 or Inferno style OS implementing a "tab as a file" style semantic.

casual-dev• 3 years ago

Question is, can I use my spare APU for this. On second thought, giving it more VRAM in BIOS only means using more RAM for it. So it would just be a RAM disk with extra steps.

Nevertheless, neat!

Taek• 3 years ago

One thing I'm not seeing mentioned is the security implications of running a filesystem in VRAM. Unless things have changed, GPU VRAM is basically free-for-all and any application can do read/write operations anywhere in the memory bank.

There are certain scenarios where this is okay but lots of scenarios where it is not (example: webgl)

londons_explore• 3 years ago

Webgl gets to read memory that isn't its own? That can't be right... There would be massive info leaks...

flohofwoe2• 3 years ago

Yeah, it isn't right. In the very early WebGL days there have been a handful of implementation bugs where previous VRAM content was accidentally exposed in textures, but these things have been fixed quickly. AFAIK, most of this is now also taken care of down on the hardware and driver level.

monocasa3• 3 years ago

Things have in fact changed. GPUs overwhelmingly have their own MMU to seperate different user contexts, including as accessed through their DMA engines. The few cases under Linux that don't such as the Raspberry PI's GPU instead use the kernel to wrap any submissions to the GPU where they can abstract and validate access to memory.

nextaccountic2• 3 years ago

How can I search for whether my GPU supports this? My GPU is a Radeon HD 8670M, on a very recent laptop

Is this the IOMMU (which isn't generally supported, except on high end hardware [0]) or something else?

I think my GPU doesn't support IOMMU

[0] https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardw...

userbinator4• 3 years ago

VRAM is accessible via an aperture in physical address space (I believe it's bank-switched for GPUs with more VRAM than the aperture size), which normally isn't directly accessible to user-mode applications, much like most hardware on Linux.

flohofwoe2• 3 years ago

WebGL goes to great lengths to not expose any 'junk' VRAM data or allow out-of-bounds access even on old GPUs. But for the last decade or so, GPUs and their drivers implement the same memory protection mechanisms as CPUs (and AFAIK WebGL was a major driving force in that area).

tenebrisalietum• 3 years ago

Use LUKS to encrypt it.

fwlr• 3 years ago

I have lived to see man-made horrors beyond my comprehension

rbanffy4• 3 years ago

That's not too bad. Most desktop work require only a tiny fraction of the memory a discrete GPU has on-board. As long as the OS manages it well enough (letting programs use the remaining memory safely while keeping the VRAM filesystem safe), seems like a good solution.

dusted• 3 years ago

Am I reading that wrong, or is it slower than 25 Gbps ethernet ?

dev_snd• 3 years ago

The readme states that currently it transfers around 2 gigabytes per second which is 16 gigabits per second.

The rest I leave as an exercise to the reader.

ptman• 3 years ago

Throughput likely, but how about latency?

redox992• 3 years ago

It's ancient hardware, and FUSE is very slow. A benchmark on modern hardware would be interesting.

rwmj• 3 years ago

It would seem more natural to map VRAM to a block device, using nbdkit. Then you could create a regular filesystem like ext4 on top. Is there a specific reason why writing an esoteric FUSE filesystem is better?

int0x80• 3 years ago

Neat! The implementation is simple, clean and well structured -- a good way to understand in more detail how a FUSE filesystem is implemented.

RobotToaster• 3 years ago

Just need to use it with a thunderbolt to pcie adapter and a graphics card, put swap on it, and you can finally have external RAM.

vatys• 3 years ago

One step away from downloading more RAM

jwilk2• 3 years ago

https://downloadmoreram.com/

redox992• 3 years ago

You can always mount an FTP server and put your swapfile there.

ZiiS2• 3 years ago

Prior art http://www.primrosebank.net/images/ZX81M64k.jpg

davidy123• 3 years ago

Wait'll you guys hear about buying more RAM and using it as a RAM disk.

doublerabbit• 3 years ago

Woah. Next you'll tell me I could increase the dimension size of my punchcard, to store more upon.

rbanffy4• 3 years ago

You can store twice as many puns on a 16" punchcard.

anon___2• 3 years ago

I have more RAM than I need, and just the other day I set up a RAM disk to put my /tmp directory on. My comic reader unpacks huge rar and zip archives into /tmp at every run, and I don't want it to wear out my SSD. I put this line in /etc/fstab:

  none  /tmp  tmpfs  defaults,size=4G  0  0

It works like a charm. And in the worst case, if I run out of RAM anyway, it readily swaps out.

bravetraveler2• 3 years ago

This is a commonly recommended thing to do! It helps IPC performance in some ways - also makes sure nothing in there persists longer than intended.

You might consider these common mount options for a little extra security:

    noexec,nosuid,nodev

(and potential headaches, to be fair)

While /tmp is a great world-writable place, these restrict it from being home to executables/devices -- common sources of abuse

LargoLasskhyfv4• 3 years ago

You could also do https://wiki.archlinux.org/title/Firefox/Profile_on_RAM , or more generally https://wiki.archlinux.org/title/Profile-sync-daemon or https://wiki.archlinux.org/title/Anything-sync-daemon

Or just skip that install to media stuff altogether, and run from some live distro booted into RAM, and running from there :-)

deathclassic• 3 years ago

hmmm. Wonder what would happen if you configured a zfs pool with this as the l2arc vdev.

LargoLasskhyfv4• 3 years ago

Would you like to when it's lacking ECC?

lostmsu2• 3 years ago

Hm, in 1 day nobody suggested this situates on the wrong abstraction level. E.g. why a file system, when it could be a virtual block device?

Vramfs – GPU VRAM based file system for Linux

Related Stories