Node.js needs a virtual file system

https://blog.platformatic.dev/why-nodejs-needs-a-virtual-file-system

by voctor • 4 months ago

275 246 4 months ago

blog.platformatic.dev

petcat2• 4 months ago

Are people still building new projects on Node.js? I would have thought the ecosystem was moving to deno or bun now

rrr_oh_man• 4 months ago

Why?

kitsune1• 4 months ago

The delusion in this comment is insane.

jitl2• 4 months ago

loud people on twitter are always switching to the new hotness. i personally can't see myself using bun until its reputation for segfaults goes away after a few more years of stabilizing. deno seems neat and has been around for longer, but its node compatibility story is still evolving; i'm also giving it another year before i try it.

_flux2• 4 months ago

Wow, I thought you were exaggerating, but no: https://github.com/oven-sh/bun/issues?q=is%3Aissue%20state%3...

Open 80, closed 492.

dzogchen2• 4 months ago

I don't really understand what the value proposition of Bun and Deno is. And I see huge problems with their governance and long-term sustainability.

Node.js on the other hand is not owned or controlled by one entity. It is not beholden to the whims of investors or a large corporation. I have contributed to Node.js in the past and I was really impressed by its rock-solid governance model and processes. I think this an under-appreciated feature when evaluating tech options.

packetlost• 4 months ago

Deno has some pretty nice unique features like sandboxing that, afaik, don't exist in other runtimes (yet). It's enough of a draw that it's the recommended runtime for projects like yt-dlp: https://github.com/yt-dlp/yt-dlp/issues/14404

zamadatix• 4 months ago

If one gets nothing from them directly, they've at least been a good kick to get several features into Node. It's almost like neovim was to vim, perhaps to a lesser extent.

pier2510• 4 months ago

I agree about the governance and long-term sustainability points but if you don't see any value in Bun or Deno is probably because (no offense) you are not paying attention.

gavmor• 4 months ago

Faster, no transpilation, dev-ex sugar.

zadikian9• 4 months ago

Note that Bun was recently acquired by Anthropic.

zadikian9• 4 months ago

Yes people are using Node.js, most likely the majority.

PaulHoule4• 4 months ago

Would be nice if node packages could be packed up in ZIP files so to avoid the security/metadata tax for small file access on Windows.

MarleTangible• 4 months ago

The number of files in the node modules folder is crazy, any amount of organization that can tame that chaos is welcomed.

koolba• 4 months ago

And if you thought malware hiding in a mess of files was bad, just wait till you see it in two layers of container files.

fmorel• 4 months ago

I remember when Firefox started putting everything into jars for similar reasons.

https://web.archive.org/web/20161003115800/https://blog.mozi...

sheept• 4 months ago

Would it work to run a bundler over your code, so all (static) imports are inlined and tree shaken?

Dangeranger• 4 months ago

There are alternative package managers like Yarn that use zip files as a way to store each Node package.[0]

[0] https://yarnpkg.com/advanced/pnp-spec#zip-access

PaulHoule4• 4 months ago

... and of course JAR files in Java are just ZIP files with a little extra metadata and the JVM can unpack them in realtime just fine.

chrisweekly2• 4 months ago

Strong recommendation to use PNPM instead of yarn or npm. IME (webdev since 1998) it's the only sane tool for stewardship of an npm dependency graph.

See https://pnpm.io/motivation

Also, while popularity isn't necessarily a great indicator of quality, a quick comparison shows that the community has decided on pnpm:

https://www.npmcharts.com/compare/pnpm,yarn,npm

MBCook• 4 months ago

It’s insane to me that node works how it does. Zip files make so much more sense, I really liked that about Yarn.

zadikian9• 4 months ago

Would accessing deps directly from a zip really be faster? I'd be a little surprised but not terribly, given that it's readonly on an fs designed for RW. If not, maybe just tar?

pie_flavor• 4 months ago

You just cat the exe with the zip file, then it is all loaded into memory at the same time on process init. This is how e.g. LÖVE does game code packaging. (It can't be tar, because this trick only works because the PKZIP descriptor is at the end of the file.)

pverheggen• 4 months ago

You can always use virtualized Linux to avoid the NTFS penalty (WSL2, VS Code dev containers, etc.)

buttsack• 4 months ago

When npm decided to have per-project node_modules (rather than shared like ruby and others) and human readable configs and library files I think the goal was to be a developer friendly and highly configurable, which it is. And package.json became a lot more than that as a result, it’s been a great system IMO.

Combined with a hackable IDE like Atom (Pulsar) made with the same tech it’s a pretty great dev exp for web devs

PaulHoule4• 4 months ago

It’s one thing or another.

Python had shared packages for a long time and those are fine up to a point but circa 2017 I was working at a place where we had data scientists making models using different versions of Tensorflow and stuff and venv’s are essential to that. We were building unusually complex systems and having worse problems than other people but if you do enough development you will have trouble with shared packages.

The node model of looking for packages in the local directory has some appeal and avoids the need for “activation” but I like writing Python-based systems that define one or more command line programs that I can go use in any directory I want. For instance, if I want to publish one of my Vite projects I have a ‘transporter’ written in Python that looks for a Vite project in the current directory and uploads it to S3, updates metadata and invalidates cloudfront and all that. I have to activate it which is a minor hassle but then I can go to different Vite projects and publish them.

moralestapia• 4 months ago

Let me be honest: a PR that size would normally take months of full-time work. This one happened because I built it with Claude Code.

The node.js codebase and standard library has a very high standard of quality, hope that doesn't get washed out by sloppy AI-generated code.

OTOH, Matteo is an excellent engineer and the community owes a lot to him. So I guess the code is solid :).

austin-cheney3• 4 months ago

Most of the 4 justifications mentioned sound like mitigations of otherwise bad design decisions. JavaScript in the browser went down this path for the longest time where new standards were introduced only to solve for stupid people instead of actually introducing new capabilities that were otherwise unachievable.

I do see some original benefits to a VFS though, bad application decisions aside, but they are exceedingly minor.

As an aside I think JavaScript would benefit from an in-memory database. This would be more of language enhancement than a Node.js enhancement. Imagine the extended application capabilities of an object/array store native to the language that takes queries using JS logic to return one or more objects/records. No SQL language and no third party databases for stuff that you don't want to keep in offline storage on a disk.

iainmerrick• 4 months ago

Why would you want a language enhancement for that, rather than just writing it in JS code? (or perhaps WASM)

duped• 4 months ago

As an aside I think JavaScript would benefit from an in-memory database.

isn't that just global state, or do you mean you want that to be persistent?

dotancohen3• 4 months ago

  > I think JavaScript would benefit from an in-memory database.

That database would probably look a lot like a JSON object. What are you suggesting, that a global JSON object does not solve?

austin-cheney3• 4 months ago

Whether it is an object, array, something else, or a combination thereof is a design decision. It is not so much about the design of the structure, which should be determined by execution performance considerations, but how information is added, removed and retrieved. Gathering one or more records from a JSON object, or array index, by value of some child property somewhere in a descendant structure of the instance index always feels like a one-off based upon the shape of the data. That could just be a query which is more elegant to read and yet still achieves superior execution performance compared to a bunch of nested loops or string of function array methods.

The more structures you have in a given application and the larger those structures become in their schemas the more valuable a uniform storage and retrieval solution becomes.

curtisblaine• 4 months ago

sorted maps with log(n) access.

Xenoamorphous• 4 months ago

Like indexedDB but in Node?

wccrawford• 4 months ago

I'm not convinced that allowing Node to import "code generated at runtime" is actually a good thing. I think it should have to go through the hoops to get loaded, for security reasons.

I like the idea of it mocking the file system for tests, but I feel like that should probably be part of the test suite, not Node.

The example towards the end that stores data in a sqlite provider and then saves it as a JSON file is mind-boggling to me. Especially for a system that's supposed to be about not saving to the disk. Perhaps it's just a bad example, but I'm really trying to figure out how this isn't just adding complexity.

TheRealPomax• 4 months ago

But then you go "hang on, doesn't ESM exist?" and you realize that argument 4 isn't even true. You can literally do what this argument says you can't, by creating a blob instead of "writing a temp file" and then importing that using the same dynamic import we've had available since <checks his watch> 2020.

notnullorvoid2• 4 months ago

There's also a module expression proposal, that would remove the need to use blob imports.

https://github.com/tc39/proposal-module-expressions

dfabulich• 4 months ago

A virtual filesystem makes it possible for the ESM you import to statically import other files in the virtual filesystem, which isn't possible by just dynamically importing a blob. Anything your blob module imports has to be updated to dynamically import its dependencies via blobs.

Normal_gaussian7• 4 months ago

    node -e "new Function('console.log(\"hi\")')()"

or more to the point

    node -e "fetch('https://unpkg.com/cowsay/build/cowsay.umd.js').then((r) => r.text()).then(c => new Function(c + 'console.log(exports.say({ text: \"like this\"}))')())"

that one is particularly bad, because umd messes with the global object - so this works

    node -e "fetch('https://unpkg.com/cowsay/build/cowsay.umd.js').then((r) => r.text()).then(c => new Function(c)()).then(() => console.log(exports.say({ text: 'oh no'})))"

phendrenad23• 4 months ago

Well there you have it.

I had to laugh, because the post you're replying to STRONGLY reminds me of this story, https://news.ycombinator.com/item?id=31778490 , in which some people on the GNOME project objected to thumbnails in the file-open dialog box because it might be a "Security issue" (even though thumbnails were available in the normal file browser, something those commenters probably should have known about, but didn't, but they just had to chime in anyway).

apatheticonion2• 4 months ago

As a user of embedded Node.js - I need the ability to package JavaScript into the binary and feed it to Node.js without writing it to disk.

My current flow is to literally embed the JavaScript in the binary, then on start, write the JavaScript code to `/tmp/{random}` and point Node.js to execute the code at that destination.

A virtualized filesystem also allows for a safer "plugin" story for Node.js - where JavaScript plugins can be prevented from accessing the real filesystem.

bsjshshsb• 4 months ago

Not a Lisp fan?

westurner• 4 months ago

Is node::vfs the new solution for JupyterLite filesystems?

From https://github.com/jupyterlite/jupyterlite/issues/949#issuec... :

Ideally, the virtual filesystem of JupyterLite would be shared with the one from the virtual terminal.

emscripten-core/emscripten > "New File System Implementation": https://github.com/emscripten-core/emscripten/issues/15041#i... :

[ BrowserFS, isomorphic-git/lightningfs, ]

pyodide/pyodide: "Native file system API" #738: https://github.com/pyodide/pyodide/issues/738 re: [Chrome,] Filesystem API :

jupyterlab-git [should work with the same VFS as Jupyter kernels and Terminals]

pyodide/pyodide: "ENH Add API for mounting native file system" #2987: https://github.com/pyodide/pyodide/pull/2987

mg• 4 months ago

    You can’t import or require() a module
    that only exists in memory.

You can convert it into a data url and import that, can't you?

afavour3• 4 months ago

What happens to relative imports?

ozlikethewizard• 4 months ago

I'm not convinced this needs to be in core Node, but being able to have serverless functions access a file system without providing storage would definitely have some use cases. Had some fun with video processing recently that this would be perfect for.

indutny4• 4 months ago

Taking the question of whether this would be a useful addition to Node.js core or aside, it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter which in my opinion is against the spirit of the project and directly violates the terms of Developer's Certificate of Origin set in the project's CONTRIBUTING.md

epolanski4• 4 months ago

Do as I say, not as I do.

On a more serious note, I think that this will be thoroughly reviewed before it gets merged and Node has an entire security team that overviews these.

indutny4• 4 months ago

As someone who was a part of the aforementioned security team I'm not sure I'd be interested in reviewing such volume of machine generated code, expecting trap at every corner. The implicit assumption that I observed at many OSS projects I've been involved with is that first time contributions are rarely accepted if they are too large in volume, and "core contributor" designation exists to signal "I put effort into this code, stand by it, and respect everyone's time in reviewing it". The PR in the post violates this social contract.

athorax2• 4 months ago

How exactly does it violate the Developer's Certificate of Origin clause?

indutny4• 4 months ago

The submitted code must adhere to either of (a), (b), (c), and separately a (d) clause of: https://github.com/nodejs/node/blob/main/CONTRIBUTING.md#dev...

If submitter picks (a) they assert that they wrote the code themselves and have right to submit it under project's license. If (b) the code was taken from another place with clear license terms compatible with the project's license. If (c) contribution was written by someone else who asserted (a) or (b) and is submitted without changes.

Since LLM generated output is based on public code, but lacks attribution and the license of the original it is not possible to pick (b). (a) and (c) cannot be picked based on the submitter disclaimer in the PR body.

digikata2• 4 months ago

Large PRs could follow the practices that the Linux kernel dev lists follow. Sometimes large subsystem changes could be carried separately for a while by the submitter for testing and maintenance before being accepted in theory, reviewed, and if ready, then merged.

While the large code changes were maintained, they were often split up into a set of semantically meaningful commits for purposes of review and maintenance.

With AI blowing up the line counts on PRs, it's a skill set that more developers need to mature. It's good for their own review to take the mass changes, ask themselves how would they want to systematically review it in parts, then split the PR up into meaningful commits: e.g. interfaces, docs, subsets of changed implementations, etc.

goalieca• 4 months ago

With AI blowing up the line counts on PRs,

Well, the process you’re describing is mature and intentionally slows things down. The LLM push has almost the opposite philosophy. Everyone talks about going faster and no one believes it is about higher quality.

dakiol2• 4 months ago

Nobody wants to review AI-generated code (unless we are paid for doing so). Open source is fun, that's why people do it for free... adding AI to the mix is just insulting to some, and boring to others.

Like, why on earth would I spent hours reviewing your PR that you/Claude took 5 minutes to write? I couldn't care less if it improves (best case scenario) my open source codebase, I simply don't enjoy the imbalance.

mixologic• 4 months ago

Worth noting that mcollina is a member of the Node.js Technical Steering Committee

everlier• 4 months ago

We call it a slip slop at work, it's ok to slip some slop if it's "our" slop :-)

kartaka83838• 4 months ago

yes this.

if there's anyone i would trust in exploring these avenues, it's him and the maintainers doing god's work in the nodejs repo in these past few years.

syrusakbary3• 4 months ago

Fully disagree with this take. Not allowing AI assistance on PRs will likely decimate the project in the future, as it will not allow fast iteration speeds compared to other alternatives.

Note aside, OpenJS executive director mentioned it's ok to use AI assistance on Node.js contributions:

  I checked with legal and the foundation is fine with the DCO on AI-assisted contributions. We’ll work on getting this documented.

[1]: https://github.com/nodejs/node/pull/61478#issuecomment-40772...

indutny4• 4 months ago

I appreciate hearing your point of view on this. In my opinion the future of Open Source and AI assisted coding is a much bigger issue, and different people have different levels of confidence in both positive and negative outcomes of LLM impact on our industry.

It is great to have a legal perspective on compliance of LLM generated code with DCO terms, and I feel safer knowing that at least it doesn't expose Node.js to legal risk. However it doesn't address the well known unresolved ethical concerns over the sourcing of the code produced by LLM tooling.

szmarczak• 4 months ago

Not allowing AI assistance on PRs will likely decimate the project in the future, as it will not allow fast iteration speeds compared to other alternatives.

It's not an AI issue. Node.js itself is lots of legacy code and many projects depend on that code. When Deno and Bun were in early development, AI wasn't involved.

Yes, you can speed up the development a bit but it will never reach the quality of newer runtimes.

It's like comparing C to C++. Those languages are from different eras (relatively to each other).

jaredklewis2• 4 months ago

AI coding is great, but iteration speed is absolutely not a desirable trait for a runtime. Stability is everything.

Speed code all your SaaS apps, but slow iteration speeds are better for a runtime because once you add something, you can basically never remove it. You can't iterate. You get literally one shot, and if you add a awkward or trappy API, everyone is now stuck with it forever. And what if this "must have" feature turns out to be kind of a dud, because everyone converged on a much more elegant solution a few years later? Congratulations, we now have to maintain this legacy feature forever and everyone has to migrate their codebase to some new solution.

Much better to let dependencies and competing platforms like bun or deno do all the innovating. Once everyone has tried and refined all the different ways of solving this particular problem, and all the kinks have been worked out, and all the different ways to structure the API have been tried, you can take just the best of the best ideas and add it into the runtime. It was late, but because of that it will be stable and not a train wreck.

But I know what you're thinking. "You can't do that. Just look at what happens to platforms that iterate slowly, like C or C++ or Java. They're toast." Oh wait, never mind, they're among the most popular platforms out there.

oystersareyum• 4 months ago

Allowing AI contributions results in lower quality contributions and allows wild things to come in and disrupt it, making it an unreliable dependency. We have seen big tech experience constant outages due to AI contributions as is...

KronisLV• 4 months ago

Not allowing AI assistance on PRs will likely decimate the project in the future, as it will not allow fast iteration speeds compared to other alternatives.

That sort of statement might also be sarcasm in another context: I personally use AI a lot, but also recognize that there are a lot of projects out there that are suffering from low quality slop pull requests, devs that kinda sign out and don't care much about the actual code as long as it appears to be running, alongside most LLMs struggling a lot with longer term maintenance if not carefully managed. So I guess it depends a lot on how AI is used and how much ideological opposition to that there is. In a really testable codebase it could actually work out pretty well, though.

lmm7• 4 months ago

Not allowing AI assistance on PRs will likely decimate the project in the future, as it will not allow fast iteration speeds compared to other alternatives.

If and when there is evidence that AI is actually increasing the speed of improvement (and not just churn), it would make sense to permit it. Unless and until such evidence emerges, the risks greatly outweigh the benefits, at least for a foundational codebase like this.

petercooper• 4 months ago

Not allowing AI assistance on PRs will likely decimate the project in the future,

I can't help but wonder if this matter could result in an io.js-like fork, splitting Node into two safe-but-slow-moving and AI-all-the-things worlds. It would be historically interesting as the GP poster was, I seem to recall, the initial creator of the io.js fork.

conartist6• 4 months ago

Pain is a signal. Even if the trick is not minding, it's still inadvisable to burn your hand on an open flame. The pain is there to help you not get hurt.

I do not think it is wise to brag that your solution to a problem is extremely painful but that you were impervious to all the pain. Others will still feel it. This code takes bandwidth to host and space on devices and for maintainers it permanently doubles the work associated with evolving the filesystem APIs. If someone else comes along with the same kind of thinking they might just double those doubled costs, and someone else might 8x them, all because nobody could feel the pain they were passing on to others

nine_k2• 4 months ago

I don't see it to be such a pain.

Bundle a full application into a Single Executable.

Embed a zip file into the executable, or something. Node sort of supports this since v25, see --build-sea. Bun and Deno support this for a longer time.

Run tests without touching the disk.

This must be left to the host system to decide. Maybe I want them to touch the disk and leave traces useful for debugging. I'd go with tmpfile / tmpdir; whoever cares, knows to mount them as tmpfs, which sits in RAM. (Or a ramdisk under Windows.)

Sandbox a tenant’s file access. In a multi-tenant platform, you need to confine each tenant to a directory without them escaping

This looks like a wrong tool, again. Run your Node app in a container (like you are already doing), mount every tenant's directory as a separate mount point into your container. (Similar with BSD jails.) This seems like the only problem that is not trivial to solve without a "VFS", but I'm not very certain that such a VFS would be as well-audited as Docker, or nsenter and unshare. The amount of work necessary for implementing that is too much for the niche benefit it would provide.

Load code generated at runtime. See tmpfs for a trivial answer. For a less trivial answer, I don't see how Node's code loader is bound to a filesystem. If it can import via https, Just use ESM loader hooks and register() your loader, assuming you're running Node ≥ 20.6.

madeofpalk2• 4 months ago

it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter

Who reviewed and approved the PR?

petetnt• 4 months ago

Personally I’d like to thank you for raising the point, it seems that tsc members are willing to ram the PR through regardless as per jasnell’s LLM analysis that honestly seems like a hostile gish galloping attempt than an actual honest analysis.

ovflowd2• 4 months ago

Matteo wrote a pretty neat article that effective counters your claim https://adventures.nodeland.dev/archive/who-is-responsible-f...

dannyfritz07• 4 months ago

Am I reading this right that Matteo is saying providence is not important because there are lots of historical cases of not having providence of code?

Many contributions contain routine, non-copyrightable material, and developers still sign off on them.
Compilers change code in ways developers do not always track. Template generators create output from their own logic. Stack Overflow answers are often copied into codebases without much thought about licensing.

Normal_gaussian7• 4 months ago

yarn pnp is currently broken on Node v25.7+;

- https://github.com/yarnpkg/berry/issues/7065

- https://github.com/nodejs/node/issues/62012

This is because yarn patches fs in order to introduce virtual file path resolution of modules in the yarn cache (which are zips), which is quite brittle and was broken by a seemingly unrelated change in 25.7.

The discussion in issue 62012 is notable - it was suggested yarn just wait for vfs to land. This is interesting to me in two ways: firstly, the node team seems quite happy for non-trivial amounts of the ecosystem to just be broken, and suggests relying on what I'm assuming will be an experimental API when it does land; secondly, it implies a lot of confidence that this feature will land before LTS.

chrisweekly2• 4 months ago

Strong rec to choose PNPM over yarn. I just posted this in a peer comment: https://news.ycombinator.com/item?id=47415173

Not spamming, not affiliated, just trying to help others avoid so much needless suffering.

zadikian9• 4 months ago

I just use npm because I like to stay as vanilla as possible. Glad that alternatives exist though.

Normal_gaussian7• 4 months ago

This is quite spammy; you could mitigate it by explaining what you think the "needless suffering" is. Having been using npm, pnpm, and yarn for many years the only benefit I find with pnpm is a little bit of speed when using the cli, but not enough that I notice; I've outlined the major yarn benefit to me 'in a peer comment' (which I didn't realise was you when I answered) https://news.ycombinator.com/item?id=47415660

I expect yarn to have a real competitor sooner rather than later that will replace it; and I do wonder if it is this vfs module that will enable it.

nwienert• 4 months ago

For many years I was using yarn with 0 issue on massive monorepos, and every year I'd hear people hyping pnpm, I'd try and switch, run into multiple bugs often open issues in pnpm itself, yes even without their link strategy, then give up and wait. After about 3 years of this I gave up and never tried again.

pamcake• 4 months ago

firstly, the node team seems quite happy for non-trivial amounts of the ecosystem to just be broken

yarn/node relations specifically are... complicated. On display on corepack (yarn project which got bundled into official nodejs distribution) issue tracker.

secondly, it implies a lot of confidence that this feature will land before LTS.

This confidence is somewhat concerning. Will it get reviewed at all or has the "trust the LLM" mandate arrived at Node too now.

notnullorvoid2• 4 months ago

I could see something like this being useful if it could be passed to workers to replace any fs access inside the worker.

mohsen12• 4 months ago

Yarn, pnpm, webpack all have solutions for this. Great to see this becoming a standard. I have a project that is severely handicapped due to FS. Running 13k tests takes 40 minutes where a virtual file system that Node would just work with it would cut the run time to 3 minutes. I experimented with some hacks and decided to stay with slow but native FS solution.

What I really want is a way of swapping FS with VFS in a Node.js program harness. Something like

     node --use-vfs --vfs-cache=BIG_JSON_FILE

So basically Node never touches the disk and load everything from the memory

Normal_gaussian7• 4 months ago

The way to do this today is to do it outside of node. Using an overlay fs with the overlay being a ramfs. You can even chroot into it if you can't scope the paths you need to be just downstream from some directory. Or, just use docker.

mohsen12• 4 months ago

making that work cross platform is pure pain

pier2510• 4 months ago

The Node team has lost the plot IMO.

By far the most critical issue is the over reliance on third party NPM packages for even fundamental needs like connecting to a database.

afavour3• 4 months ago

What would a Node-native database connection layer look like? What other platforms have that?

Databases are third party tech, I don’t think it’s unreasonable to use a third party NPM module to connect to them.

ksherlock• 4 months ago

Perl has DBI. PHP has PDO.

mike_hearn2• 4 months ago

Most obviously, Java has JDBC. I think .NET has an equivalent. Drivers are needed but they're often first party, coming directly from the DB vendor itself.

Java also has a JIT compiling JS engine that can be sandboxed and given a VFS:

https://www.graalvm.org/latest/security-guide/sandboxing/

N.B. there's a NodeJS compatible mode, but you can't use VFS+sandboxing and NodeJS compatibility together because the NodeJS mode actually uses the real NodeJS codebase, just swapping out V8. For combining it all together you'd want something like https://elide.dev which reimplemented some of the Node APIs on top of the JVM, so it's sandboxable and virtualizable.

pier2510• 4 months ago

Bun provides native MySQL, SQlite, and Postgres drivers.

I'm not saying Node should support every db in existence but the ones I listed are critical infrastructure at this point.

When using Postgres in Node you either rely on the old pg which pulls 13 dependencies[1] or postgres[2] which is much better and has zero deps but mostly depends on a single guy.

[1] https://npmgraph.js.org/?q=pg

[2] https://github.com/porsager/postgres

beart• 4 months ago

Outside of sqlite, what runtimes natively include database drivers?

pier2510• 4 months ago

Bun, .NET, PHP, Java

nulltrace2• 4 months ago

I publish a package with zero deps and people still pull in a pile of transitive stuff from their lockfile. "pg" has 13 dependencies and nobody even blinks. One gets compromised and suddenly every Node backend using Postgres is in scope. Bun shipping native drivers feels like the right call, fewer moving parts.

NoNameProvided• 4 months ago

I understand the general point you're making, but the pg package isn’t a good example. It has 6 deps, not 13, and 5 of those are internal packages from the same monorepo without additional dependencies. There’s only a single external dependency, and that one brings in just one additional package.

In my opinion, the pg repo and packages are an example of how OSS stuff should be maintained. Clean repo, clean code, well-maintained readme, and clearly focus on keeping things simple instead of overcomplicating.

adzm2• 4 months ago

How does electron do this with its packaged files? I suppose it does not work with module resolution?

lacoolj2• 4 months ago

Using Claude for code you use yourself or at your own company internally is one thing, but when you start injecting it into widely-shared projects like this (or, the linux kernel, or Debian, etc) there will always be a lingering feeling of the project being tainted.

Just my opinion, probably not a popular one. But I will be avoiding an upgrade to Node.js after 24.14 for a while if this is becoming an acceptable precedent.

atomicnumber3• 4 months ago

I still think everyone is trying to run away from the copyright problems with AI, and suspect it's going to come back to bite them. Eventually. (No I'm not willing to bet on exactly when because I'm sure it'll be a lot longer than I'd like).

bronlund• 4 months ago

Yeah. That’s what we need. More Node.

gwbas1c• 4 months ago

Can you dynamically load code via eval?

(I know, I know, it's ugly and has its own set of problems)

torginus• 4 months ago

Why do people keep reinventing OS features?

There's Docker, OverlayFS, FUSE, ZFS or Btrfs snapshots?

Do you not trust your OS to do this correctly, or do you think you can do better?

A lot of this stuff existed 5, 10, 15 years ago...

Somehow there's been a trend for every effing program to grow and absorb the features and responsibilities of every other program.

Actually, I have a brilliant idea, what if we used nodejs, and added html display capabilities, and browser features? After all Cursor has already proven you can vibecode a browser, why not just do it?

I'm just tired at this point

ryandrake• 4 months ago

One of the worst is media players that all insist on grafting their own "library" on top of my already-working OS filesystem. So I can't just run the media player and play files. No, that would be too simple. I have to first "import" my media into a "library" abstraction and then store that library somewhere else on my filesystem. Terrible!

SAI_Peregrinus• 4 months ago

There's a legitimate problem they're trying to solve there: there are several ways to sort media that don't match up well with a hierarchical filesystem¹. They solve it badly. Good players maintain a database for efficient queries of media metadata, and periodically rescan the folders to update it. Shitty media players try to manage the files themselves, and still end up needing to maintain a database. The worst of these use the database to manage the contents of their storage files (or store the files themselves in the database), if something isn't in the database they delete the files. Adobe Lightroom Classic does this, if your database gets corrupted it deletes all your RAW files!

¹E.g. if you've got music, and it's sorted `artist/album/track<n>.extension`, and two artists collaborate on an album, which one gets the album in their folder? What if you want to sort all songs in the display by publication date? Even if they use the files on your filesystem without moving them, some sort of metadata database will be needed for efficient display & search.

williamstein• 4 months ago

This exact thing solves a huge problem with SEA binaries as he points out in his post. You can include complicated assets easily and skip an ugly unpack step entirely. This is very useful.

themafia• 4 months ago

You can’t import or require() a module that only exists in memory.

Sure you can. Function() exists and require.cache exists. This is _intentionally_ exploitable.

verdverm2• 4 months ago

Separate the valid critiques on other comments, Go's io.FS interface is really nice for making these sorts of things. Is there something like this in Node already? (with base implementations like host and in memory)

dabbz2• 4 months ago

Another thread about Edge.js has some similar concepts that might be applicable? They're not 1:1 the same but it involves Sandboxing https://news.ycombinator.com/item?id=47417398

verdverm2• 4 months ago

I would put virtual or filesystem abstractions in a different category than sandboxing, which puts restrictions over the virtual or native implementations.

sidewndr46• 4 months ago

Don't all projects eventually grow to encompass service discovery?

gnarbarian• 4 months ago

one of the reasons I prefer deno is the availability of indexeddb (and all the other great stuff that comes with it out of the box)

butz• 4 months ago

How about trying to reduce dependencies? 11ty is going in correct direction, dropping significant chunk of various dependencies or replacing them with packages with no dependencies or using platform features, that becomes readily available.

pier2510• 4 months ago

Not a priority for the Node team, unfortunately.

giancarlostoro2• 4 months ago

I pointed the AI at the tedious parts, the stuff that makes a 14k-line PR possible but no human wants to hand-write: implementing every fs method variant (sync, callback, promises), wiring up test coverage, and generating docs.

This is the biggest takeaway for me for AI. It's not even that nobody wants to do these things, its that by the time you finish your tasks, you have no time to do these things, because your manage / scrum master / powers that be want you to work on the next task.

Lerc• 4 months ago

I think the insight there is that the increased productivity of AI could be used to add features where the end results are weighing the ability of the AI against the ability of an individual implementing the same thing.

The alternative is that you work on the same number of features and utilize the ability to make those features as robust as you know they could be, but you have other pressing matters to attend to. That's weighing the ability of AI against the ability of neglect.

Culonavirus• 4 months ago

That's perfectly understandable. But has no business being in a large open source project, let alone world class one like Node or (god forbid) the Linux kernel. Get that shit the fuck out.

potsandpans• 4 months ago

Get that shit the fuck out.

No.

minraws• 4 months ago

Why is this not a library what is this insanity??

socalgal2• 4 months ago

What's special about node.js here? Does golang, C#, python, ruby, java, etc... have a virtual file system?

I get it, I've implemented things for tests, I'm just wondering if this shouldn't be solved at an OS level.

--- update

Let's put this another way, my code does effectively, child_process.spawn('something-that-reads-and-write-a-file')

now I'm back to the same issue. To test I need a virtual file system. Node providing one won't help.

sauercrowd2• 4 months ago

Go actually does https://pkg.go.dev/embed

I do think it's more painful to distribute files when you're a distributed as a single binary vs scripts, since the latter has to figure out bundling of files anyway.

But still - it does exist

benatkin3• 4 months ago

Embed is read-only at runtime. This proposed vfs module for Node.js is a full virtual file system.

nine_k2• 4 months ago

Zip files are created in such a way that they can be a part of an executable file. (This is how self-extracting archives used to work.) Support for reading zip files is lightweight, and is present almost everywhere.

A ZIP fork embedded into the executable should be an obvious read-only VFS implementation. Bring your assets with you, even maybe build them with the standard zip utility.

It should take relatively few LOCs, provided that libzip is already linked into the executable anyway.

paradox460• 4 months ago

In the past, I've implemented this in Ruby and elixir on Linux systems using a ramdisk.

antonvs• 4 months ago

Java, Haskell, Rust, and several other languages do have solutions for this. Here’s one for Rust: https://docs.rs/vfs/latest/vfs/

Retr0id• 4 months ago

Python doesn't have a virtual filesystem in general but it is possible to shim the import mechanism.

cmrx64• 4 months ago

this is a pretty bad vfs. there are pure “cap manifest” approaches that don’t pull in decades of cruft semantics. don’t build systems that aren’t objectstore native in 2025 (since this work was initiated in december).

il-b• 4 months ago

no human wants to hand-write: implementing every fs method variant (sync, callback, promises), wiring up test coverage

That’s so dehumanizing, I would happily write such code.

chmod775• 4 months ago

Yes, but no. Node itself merely needs a standardized, pluggable layer of indirection in its file APIs. If someone wants to implement a VFS using that, that's cool.

Basically an "fs-core" that everything ultimately goes through, and which can be switched out/layered with another implementation. Think express-style routing but for the filesystem.

That'll keep things simple in node's codebase while handing more power to users.

keepamovin• 4 months ago

The way I bundle into SEA is modules that need to be imported from disk (that can't be bundled due to node or wasm modules), is just include them in the assets, and do a "write to tmp, import, delete" flow. It works.

Not saying vfs is bad, just it's not impossible in a few lines of code to set up that. My idea for a simple version of a vfs in node is to use a RAM disk/RAMfs - would that work?

philo23• 4 months ago

Little bit saddened the sqlite provider doesn't use the SQLite archive format under the hood. Seems like it'd be a good fit for what they're trying to achieve + give you an easy way to create/extract the files out of the virtual file system.

The sqlar schema is missing some of the info thats being stored atm, but there's nothing stopping you from adding your own fields/tables on top of the format, if anything the docs encourage it. It is just a sqlite database at the end of the day.

https://www.sqlite.org/sqlar.html

AndyKelley• 4 months ago

I used Node.js extensively for about a decade, both professionally and as a hobby. I don't buy the problem statement.

These arguments don't even make sense, they look LLM generated. I can't even formulate a disagreement against this nonsense.

est• 4 months ago

This brings back memories back in the days I mess around Resource Hacker for Win32 EXEs.

I miss those days where you can tweak all kinds of software GUI by your self. Change icons, menus, shortcut keys, etc.

AntonCTO• 4 months ago

It's not Node.js that needs a virtual file system. It's JavaScript.

huksley• 4 months ago

There is already memfs package which implements virtual fs, with other packages as well. What we need is to support import/require working with that vfs.

ThomIves• 4 months ago

I loved this notion from just the title, and then eagerly read the post. BRAVO! I am investigating if I can, and how to, exploit this ASAP.

Node.js needs a virtual file system

Related Stories