Parsing the .DS_Store File Format (2018)

https://0day.work/parsing-the-ds_store-file-format/

by popcalc • 3 years ago

150 84 3 years ago

m348e912• 3 years ago

>Apple's operating system creates this file in apparently all directories to store meta information about its contents. In fact, it contains the names of all files (and also directories) in that folder.

Anyone have an idea why modern versions of MacOS still uses a .DS_Store file? The OS could just as easily read and display the contents of a directory. And deleting a .DS_Store file doesn't seem to have any noticeable negative affect.

Lt_Riza_Hawkeye• 3 years ago

I believe if you've dragged some files around in that folder, it stores where those were. Or if you've set the folder to sort by name or date modified, etc.

russellbeattie• 3 years ago

I've noticed that the .DS_Store file doesn't appear in a directory until I've clicked the triangle in a list view to see the contents of a subfolder. Then it creates one, I assume, to keep track of which subfolders are open.

MBCook2• 3 years ago

Right. I would assume custom icons and labels (do those still e its?).

Basically it seems to be the replacement for the old resource fork data on non-Apple file systems.

kitsunesoba2• 3 years ago

Yeah, it basically maintains Finder window state.

One thing I've wondered is if this method still makes sense on modern hardware compared to keeping all that information in a centralized sqlite file or something along those lines. I could see individual files being advantageous on a slow mechanical HD where seeking through a file that could eventually grow large might pose issues, but that shouldn't be a problem on an SSD.

The main advantage I can think of that might still be relevant today is that it makes Finder window state machine-independent, so when you open a folder on two different machines it'll still open in list mode sorted by Date Modified or whatever.

CharlesW• 3 years ago

Anyone have an idea why modern versions of MacOS still uses a .DS_Store file?

The Wikipedia article is a pretty good resource for this: https://en.wikipedia.org/wiki/.DS_Store

gorgoiler• 3 years ago

If you know what a dotfile is, it’s there for the same reason.

(Dotfiles are per-user system configuration options written as a file in the user’s home directory. This means that for any system on which that home directory exists, that user’s configuration options will be applied. If the home directory is shared across multiple computers over a network, or if it is backed up from one computer and restored to another, then that user’s settings will persist.)

If these settings were stored in some other central database then if you copied the macOS folder from one computer to another, your copy tool would have to know how copy the settings as well. Keeping those settings inside the folder itself means that your copy tool only needs to know about copying folders, instead of copying folders and also copying settings.

It’s one of those Unix principles where everything is treated as a file: an elegant technique for a more civilized age.

dreamcompiler3• 3 years ago

.DS_Store is not part of HFS+ or APFS. It's a kludge to give the OSX Finder a place to store its own metadata.

MacOS prior to X had much more elegant solutions to the problem. To this day, when you switch to/from icon/list view in the Finder, more windows than just the current one switch. In MacOS 9 that loosey-goosey unpredictable behavior never happened.

illiarian• 3 years ago

To this day, when you switch to/from icon/list view in the Finder, more windows than just the current one switch

And there's almost no way to make this behavior stick because... it's different if you click through folders and if you click on pinned folders in Finder sidebar.

I mean... How?!

GeekyBear• 3 years ago

Anyone have an idea why modern versions of MacOS still uses a .DS_Store file?

For the exact same reason Windows has a Desktop.ini file?

They store per directory settings for the Mac Finder and Windows Explorer.

alin232• 3 years ago

The file is used for storing layouts (show as icons/list/grid etc.) and various settings (sorting, grouping etc.).

Storing a local file inside the directory allows for moving that directory around while also keeping those settings.

The file is invisible for users of Finder so it is still the best solution for this task.

I have a .DS_Store line in a global .gitignore and don't notice it much but when I'm annoyed by DS_Store files in a specific directory, I usually bulk delete them using `fd`:

    fd -u --type file --fixed-strings .DS_Store $HOME/Projects/some-file-server-project -x echo rm {}

gcr• 3 years ago

i'm surprised that Apple doesn't do directory-level metadata directly in the filesystem itself, e.g. with xattrs on the directory inode.

SOLAR_FIELDS• 3 years ago

Curious what creative use cases people have around this bizarre file format. The only reason myself and millions of other people have to deal with this stupid file is adding it to .gitignore as part of the ritual of spinning up a new codebase

pilif5• 3 years ago

nitpick: As it’s a platform and machine specific thing, technically it shouldn’t be in .gitignore but in .git/info/exclude

People working on Linux or Windows do not need to know about garbage your dev env leaves on your machine.

I’m saying this half-seriously as I fully understand that these are so common, it’s more convenient to have the exclusion synced between clones.

That said, as a purist, none of the repos I’m watching over have .DS_Store in their gitignore

MBCook2• 3 years ago

Also there is a preference you can set with the defaults command to stop MacOS from putting them on file systems it encounters.

dopidopHN• 3 years ago

Interesting, I guess I live in impur lands :)

I understand the pollution concern… but unless that .git/info/exclude is standard in the company, what prevents a intern to not have a correct exclusion there, and happily push a .ds file ?

.gitignore looks more robust to the obvious user. And it’s not like it’s a file you need to look at attentively every day.

But yes, I do see that you are correct!

monkellipse• 3 years ago

An informative nitpick, thank you.

charcircuit• 3 years ago

By that logic it should be in your global excludes file. It practice it will save more time if you just add it to your project's .gitignore instead of wasting time having everyone else configure their system. O(1) amount of work compared to O(n)

WanderPanda• 3 years ago

Interesting! The kind of answer ChatGPT can not produce (afaik) because it is too far from the mean mode of how the world is operating

chrisfosterelli• 3 years ago

I think this depends on how you interpret it. All the docs[0] say are:

Patterns which should be version-controlled and distributed to other repositories via clone (i.e., files that all developers will want to ignore) should go into a .gitignore file.

and

Patterns which are specific to a particular repository but which do not need to be shared with other related repositories (e.g., auxiliary files that live inside the repository but are specific to one user’s workflow) should go into the $GIT_DIR/info/exclude file.

To me it seems obvious that all developers would want to ignore .DS_Store even if only some of them would generate it. And certainly you want it distributed to other mac users when they clone. Further, it's certainly not the case that .DS_Store is specific to just one user's workflow. On all three cases therefore gitignore would be reasonably appropriate.

[0] https://git-scm.com/docs/gitignore

jillesvangurp• 3 years ago

As long as that isn't set up by default, you can't rely on that being setup correctly for everyone or even most people. So .gitignore should include the file regardless of that setting.

account423• 3 years ago

I agree that platform-specific files shouldn't be in .gitignore, but it does make sense to put .* there and then only allow specific .files that you actually want to check in.

50• 3 years ago

The only reason myself and millions of other people have to deal with this stupid file is adding it to .gitignore as part of the ritual of spinning up a new codebase

I stole this from somewhere: Create a .gitignore_global file and include .DS_Store, then in the [core] of your .gitconfig include excludesfile = /Users/janedoe/.gitignore_global.

whalesalad• 3 years ago

You can ignore it in your global git config, too. Once per host.

rjmunro• 3 years ago

Something like:

git config --global core.excludesfile ~/.gitignore-global echo .DS_Store >> ~/.gitignore-global

tannhaeuser• 3 years ago

Having .DS_Store all over the place sucks, but to be fair so does having other dot files, such as .git which is an entire subdir even. At least the convention of hiding dot files from file listings is uniformly observed.

account423• 3 years ago

The difference is that .git is only in directories that you explicitly set up as a git repository whereas .DS_Store, Thumbs.db, .desktop and similar get barfed wherever by the file manager without doing anything that a user would consider as a modification to the directory.

planede• 3 years ago

Arguably if you work on a mac and you use git regularly then just add it to ~/.config/git/ignore, and it will be ignored everywhere.

srhtftw• 3 years ago

I instinctively delete these .DS_Store files whenever I see them but your "creative uses" comment makes me think given common .gitignore practice this might be a great place for AWS keys or whatever you never want checked into VCS /s

lapcat3• 3 years ago

(2018)

saagarjha3• 3 years ago

Added, thanks

lapcat3• 3 years ago

Did you add that? And if so, how?

mdmglr• 3 years ago

Related: If you change the folder icon, Finder will store an "Icon?" file in that folder.

More info here: https://superuser.com/a/298798

My question is: Linux and Mac both have the ability to store key-value pairs on files/folders with xattr. Why did or does Apple continue to use .DS_Store?

dreamcompiler3• 3 years ago

Good question. Possibly because that approach wouldn't work on readonly files. (But in the case of readonly directories .DS_Store wouldn't work either).

Possibly because not all copy programs copy metadata.

Possibly it's just technical debt: .DS_Store was added to MacOS before xattr and rewriting the Finder to use xattr is not high priority.

flomo• 3 years ago

Windows too with NTFS attributes. The answer is baseline FAT compatibility.

But at the very least, they could set the FAT hidden attribute on this stuff. It must be some mentality like "Well, I'm not littering in my own yard?"

ehPReth• 3 years ago

while we're here discussing what files should be pre-emptively added to your web server's deny entry; what else should be in there.. besides, say, .git?

I guess you could block .* except .well-known 'just in case'

Dalewyn• 3 years ago

As a Windows wizard, desktop.ini and thumbs.db both come to mind.

tetha• 3 years ago

Shouldn't you rather deny all, and carefully carve out paths you need to allow?

pabs3• 3 years ago

Other resources for this file format:

http://fileformats.archiveteam.org/wiki/Desktop_Services_Sto...

eviks• 3 years ago

Is there any chance for these dropping to go away and be replaced by a proper database of whatever Finder info is in there?

thrdbndndn• 3 years ago

It's not exactly the same thing, but Windows used to have a thumb.db to store, well, thumbnails, in each folder that has thumbnail-able files.

I hated it, most of people hated it. MS finally got rid of it (in Vista or 7) and store thumbnails in a centralized place.

However, I now start to miss it.

I store lots of large image files (think 1200 dpi PNG scans) on my computer, and it took Windows years to create thumbnail for them, while locking the entire hard disk (and sometimes the whole explorer.exe) in the meantime. This process, while obviously is much slower on spinning HDD, it surprisingly isn't really that fast on NVME SSD, either.

This would be fine if it's a one-time process, but unfortunately the aforementioned centralized thumbnail cache either have an expiration time, or have a size limit (or both), so the reality is if I didn't visit a folder for awhile, its thumbnails are all gone, so they have to be generated again, make the explorer unresponsive for some time again.

I have been annoyed by this for long time but I don't have any solution. The best workaround is to only view such folders in list/detail mode, and rely on 3rd party image manager (I use XnView) to show the preview/thumbnail -- which has a much faster thumbnailing speed and more robust thumbnail cache management.

simplotek• 3 years ago

This process, while obviously is much slower on spinning HDD, it surprisingly isn't really that fast on NVME SSD, either.

Windows is renowned for having awful performance in usecases involving accessing many small files even for read-only applications. I know a team that even went as far as testing switching away from Windows to build Windows apps and instead switched to cross-builds on Linux just to avoid that performance sinkhole. Meanwhile I worked on a multiplatform project where macOS builds on a Mac mini for the exact same project took less than half the time to build than on a beefed up Windows workstation.

Here's an interesting discussion on HN from a couple of years ago on this topic.

https://news.ycombinator.com/item?id=18783525

kevin_thibedeau• 3 years ago

It doesn't help that corporate Windows machines are infested with AV snakeoil scanners slowing down FS access.

kakwa_• 3 years ago

This article makes me wonder: what tools do you use when reversing a custom file format?

Personally, having done so recently to reverse a custom game archive format, I have only used the usual hexdump, strings, grep, sed, etc. I'm wondering if there are more powerful tools to do this kind of job?

eXpl0it3r• 3 years ago

You usually want something for which you can define a structure and parse the file with it. There are a bunch of commercial and free tools, I've used the following with success in the past:

- 010 Editor (commercial)

- Kaitai Struct (free & open source)

Also has the advantage of directly generating parsers from the defined structures.

dancemethis• 3 years ago

...It's _not_ from the Nintendo DS?

_ZeD_• 3 years ago

Is there a reliable way to make the os NOT create these `.DS_Store` files? I really don't like them and the fact that they pollute every folder.

I'm on Big Sur (11.7.1) and I didn't find a way to make finder NOT create them.

It's (one of the many) irritating idiosyncrasies of this OS

lucideer• 3 years ago

I'm not aware of any reliable way, but anecdotally I've found that they seem to be mostly triggered by Finder usage, rather than the OS itself. I've gotten out of the habit of using Finder (instead mostly just use terminal or file-hierarchy UIs in whatever relevant app/IDE), and it's a long time since I've had a .DS_Store issue.

alin232• 3 years ago

I don't think there's any way to do that, the files are needed by Finder to store per-folder layouts and settings.

You can however create a cronjob that deletes the files as often as you want, the operation takes 11 seconds on my 1TB SSD with a fairly large home folder:

    fd -u --type file --fixed-strings .DS_Store $HOME -x echo rm {}

obviously, remove the echo to actually delete the files

ryandrake• 3 years ago

I don't think there's any way to do that, the files are needed by Finder to store per-folder layouts and settings.

I've always hated this excuse from software developers (I know it's not your excuse, alin23).

User: How do I stop my computer from doing Unwanted Thing X?

Dev: You can't. Unwanted Thing X is needed so we can do Unwanted Thing Y.

Fucking A! Stop doing unwanted things on my computer! My computer should only be creating files I command it to create. It shouldn't be doing things I don't want it to do simply because it conveniences a developer living 1000 miles away in Silicon Valley. My filesystem is there to store my files not some developer's metadata. It should not contain this trash from my operating system. At least give me a DONT_CREATE_THIS_DS_STORE_CRAP environment variable or System Preference to set!

I'll be generous and grant one root directory tree to the OS for its litter. Linux can have /var, macOS can have /System and maybe /Library too. Windows can have C:/WINDOWS. Everywhere else in my OS, hands off, please!

plorkyeran• 3 years ago

I don't want finder to store per-folder layouts and settings. I don't even care about the .DS_Store files very much. I just don't want the functionality that they exist to enable.

nielsbot• 3 years ago

I think the .DS_Store file is in the classic macOS resource file format: https://dev.os9.ca/techpubs/mac/MoreToolbox/MoreToolbox-99.h...

Parsing the .DS_Store File Format (2018)

Related Stories