We Are Doing Files Wrong (2021)

https://simonsafar.com/2021/we_are_doing_files_wrong/

by Expurple • 1 year ago

5 4 1 year ago

jll29• 1 year ago

That's one of the few times I've read about a proposed innovation "in the spirit of UNIX" that was not already present in the original UNIX or one of its descendants.

  UNIX: Everything is a file.
  => A directory is a file.

  Parent post: Everything is a directory.
  A file is a directory.

I.e., a switch from "There are files and special files called directories that are handled differently." to the recursive definition "There are files, which are made up of 0..n files (blobs) and 0..n subdirectories" - so file versus directory is just a VIEW.

Makes sense & would make writing traversal code for files wiht internal structure much easier to read and write.

Expurple3• 1 year ago

to the recursive definition "There are files, which are made up of 0..n files (blobs) and 0..n subdirectories"

I think, it's more like "a file node contains metadata, a binary blob of data (may be empty), and 0..n child files".

Agreed that this idea is very elegant and removes special cases, nodes become uniform. And the argument for reusing the OS(FS)-provided tree abstraction is compelling.

Although, I can imagine some performance concerns in the real world. If implemented naively and similarly to the existing Unixes, this model results in a lot of small fragmented blocks and separate syscalls+descriptors for dealing with each small file. Also, when the "tree" is actually a sequential array of nameless elements, there's some extra overhead involved with writing and storing made-up file names, as well as sorting by name when reading. This could be remedied by some new API. And a single tree implementation reused by everything could be more cache-friendly than having a userland parser for every "old" format in every application.

Anyway, this mental model is useful and I'd like to see and try out the "automounting" that the author describes.

Expurple3• 1 year ago

Can't edit the parent comment anymore, so I'll append my other thoughts here.

I remembered that the "automounting" already exists in some forms, and I really like these instances. When you click on an archive in a good file manager, it opens a "folder" view with the archive contents. The difference between an archive and a folder is arbitrary. It shouldn't exist and only complicates things for everybody. I assume that many applications today hand-code the logic for "if the user drags and drops a folder, we need to zip it before sending". Or they don't, and the user has to zip manually :)

One could say that storing program data as "transparent" folders instead of "opaque" binary files is too much detail for the user. And users can accidentally damage something (e.g. delete one of the files inside) more easily. But I have a few counter-points:

1. Many applications already use folders, but they're doing fine.

2. File managers already open many filetypes in an editor by default (e.g. plain text, office docs), but these files are doing fine.

3. A good file manager should recognize most file types and do the more reasonable and safe thing. If JPEG is reimplemented as a folder, clicking on a JPEG should still open the image viewer instead of the folder view. That's already the case with Mac OS app bundles (the example from the original post).

4. Actually, now that folders have a "data" field, it can be used for storing arbitrary metadata. This is extremely powerful and can be used for HIDING extra details from casual users! E.g. there could be a standartized metadata header that hints to the file manager that it should treat the folder as an "opaque" file and not show the user the contents. Now, old applications that already used folders for their internal state, can mark these folders as "opaque" and prevent casual users from messing with the contents! While still allowing to see, move and delete the folder as a whole (unlike hidden folders). And while providing uniform FS access to applications and advanced users.

Man, I really like this idea of arbitrary metadata for folders... It's not as necessary for files, because in practice you can just put the "metadata header" in the beginning of the main "data" (as many file formats do).

Expurple3• 1 year ago

I can imagine some performance concerns in the real world.

A friend has pointed out that, for performance reasons, games already tend to sidestep the filesystem and bundle everything into container/archive files, like .pak [1]. It also reminded me of how compile times are often noticeably faster on Linux vs Windows, because its filesystem is better optimized for handling many small files.

Honestly, it's weird and makes me kind of sad. Filesystems are such an important, convenient and (mostly) standardized and portable abstraction. But to this day, it's often a bottleneck and too slow for some domains. It seems like there's a lot of missed opportunity here.

when the "tree" is actually a sequential array of nameless elements, there's some extra overhead involved with writing and storing made-up file names, as well as sorting by name when reading. This could be remedied by some new API.

As I keep thinking about it, the fastest approach is still a sequential binary blob. It avoids indirection, fragmentation and needlessly storing separate metadata for each element. A VFS mount could still be implemented on top, for accessing elements through a filesystem interface (something like `my_array_file/0`, reporting the same medatata as the parent).

But if storing separate FS-level metadata for each element is actually desirable and indirection+fragmentation isn't a critical problem, we could use folders for arrays. As a partial optimization, we could introduce a special "ordered" folder type. It requires explicitly ordering the subfiles on write, so that later listing the children is sorted and fast by default.

Option 1. The ordering is a separate metadata in the folder. Subfiles still have unique names that are not related to the ordering. This could be useful for the app, or could be unnecessary. POSIX apps (that don't know about "sorted folders") would still re-sort the subfiles by name and could get a different ordering as the result.

Option 2. The subfiles don't have user-provided names. For POSIX-compatibility, the VFS supports "artificial" file names like 0000000, 0000001, 0000002 (the number of digits according to the FS limits, sorts as text correctly) or 0, 1, 2 (prettier, independent of the FS limits and portable, but doesn't sort correctly as text).

In some sense, this special folder type is against the spirit of the original article. It has special write restrictions that prevent treating every inode the same way (as an arbitratily writetable folder). But it's still in the spirit of the original in the sense of reusing the standard FS abstractions and features as much as possible.

[1] https://quakewiki.org/wiki/.pak

We Are Doing Files Wrong (2021)

Related Stories