.Trashes, .fseventsd, and .Spotlight-V100

http://blog.hostilefork.com/trashes-fseventsd-and-spotlight-v100/

by pmoriarty • 11 years ago

89 66 11 years ago

blog.hostilefork.com

konsumer• 11 years ago

I do this if I forget to add .DS_Store to .gitignore when working with git repos: find . -name .*DS_Store -delete

singlow2• 11 years ago

Just be careful. A coworker of mine did this recently and forgot the -name part. Accidentally deleted everything, including the .git directory. He sure wished he had pushed that day.

manicdee3• 11 years ago

*that hour.

Surely you don't cruise through to the end of a day without pushing at least once before lunch? :(

manicdee3• 11 years ago

    git config --global core.excludesfile ~/.config/git/ignore

    cat >> ~/.config/git/ignore
    .DS_Store
    .AppleDouble
    .LSOverride
    .AppleDB
    .AppleDesktop
    .apdisk
    .fseventsd
    .Trashes
    .Spotlight-V100
    ._*
    Icon
    Network Trash Folder
    Temporary Items
    Icon

arnehormann• 11 years ago

you have missed

mkdir -p ~/.config/git

xorcist• 11 years ago

Why not .* ?

I can't think of a good reason to commit hidden files.

kayoone• 11 years ago

put stuff like this in your global gitignore and you will bever have to worry about these files again. Stuff like this does not belong in a projects gitignore anyway because it's system specific.

pippy• 11 years ago

It is annoying when copying a movie to a USB, and having to empty the entire trash bin to fit it on the USB stick. I guess it does enforce you to be a bit cleaner.

I don't really care about .DS_store etc as they're hidden, and thumbs.db is just as bad.

simlevesque• 11 years ago

I don't really care about .DS_store etc as they're hidden

They ain't if you are on Windows.

derefr4• 11 years ago

That's really just silliness on the part of Apple; if they're creating dotfiles on a FAT32/NTFS/ExtFS disk, they should be setting the Hidden attribute for those same files. This is what e.g. Samba does automatically for network-attached FAT/NTFS folders on OSX/Linux clients, if I recall.

recursive• 11 years ago

Thumbs.db doesn't end up in every zip file ever.

nkantar• 11 years ago

This accomplished two things:

1. I now know a few things I've wondered for a long time.

2. I have just been reminded of how little I really know, even about devices I use for most of my waking hours.

derefr4• 11 years ago

A few little arguments in favor of this practice:

1. You plug a USB device into one Mac. It asks you if you want to use it for Time Machine. You say no. Now, no other Mac will ask you this question again—because the preference has been stored on the drive, instead of on the computer.

2. You create a folder with an app you've written in it, and customize it with a cute "drag [this] onto [shortcut to Applications folder]" background, centering the icons on the background positions and sizing the window to perfectly fit the representation. You then copy this folder to another Mac (or any equivalent process, such as converting it into a DMG disk image and having someone download it.) They see exactly what you saw.

3. You keep an external hard disk that you frequently move back-and-forth between two Macs. The first time you plugged it into one of the Macs, it was indexed. Each time you modify it on either Mac, that index gets updated. The most recently created files are the most likely to be the ones you'll want to find through Spotlight when you plug the disk into the other Mac—but if the index was machine-local rather than disk-local, the new Mac wouldn't know about the new files yet, and would have to finish re-indexing to find them. (This is, of course, what happens when you modify the disk on a Windows computer, and it sucks. We really need a "metadata index API" to start being an expected offering of a filesystem, such that all OSes interacting with the filesystem can read and update it!)

On the other hand, Windows' Thumbs.db is exactly the sort of thing you don't want cached on the media. It'd be a great innovation if, say, cameras could pre-generate Thumbs.db files so Windows could immediately show the user what was on an SD card the first time it got plugged in... but cameras don't, because Thumbs.db is a random proprietary format. So, instead, it's only useful in the same situation that the Spotlight index is: moving a drive full of photos back and forth between two computers. Which might have separate screen DPI, and thus be set to show different thumbnails, meaning new thumbnails will get generated anyway.

TylerE• 11 years ago

I've always wonders why apple doesn't have some sort of .app.gz "special". Why does DMG have any reason to exist?

Even if you really really think you need to ship a readme (Hint: No one reads them, and you don't) a zip is simpler and opens quicker.

jordigh3• 11 years ago

Why does DMG have any reason to exist?

History and backwards compatibility.

The format itself also does a bit more than just a file archive, as it's a disk image format. You can do things like style the background and so forth.

derefr4• 11 years ago

1. Because a DMG can hold an HFS+ filesystem, and an HFS+ filesystem can represent files with resource forks. So, backward compatibility.

2. Because a DMG actually is a disk image, not a partition image—and therefore can represent a collection of volumes. If I recall, OSX's current "clean" install process effectively just takes a DMG containing {EFI, OS Base, Full Restore Image} volumes and uses the functionality of the "Erase and Overwrite with a DMG" option in Disk Utility to replace your disk's contents with it.

flomo• 11 years ago

why apple doesn't have some sort of .app.gz "special"

In addition to other fine explanations provided, it did. The defacto standard was ".sit.hqx" -- BinHexed Stuffit archive, both third-party tools that survived into the OS X era. Disk Images came about when Apple finally realized the pain of delivering Mac apps over the internet.

Also, since I'm posting, OS X should really set the H attribute on these dot files when writing to a FAT drive, rather than relying on Unix shell convention.

ploxiln2• 11 years ago

Or, OS X could wait until the appropriate time to offer you the option of messing with your disk.

1. Don't ask about using the disk for Time Machine. When the user goes to Time Machine because they want to use Time Machine, then have a button in the sidebar "Use drive blah for Time Machine" for any non-time-machine external drives present.

2. Have a button at the top of a finder window in which icons have been re-arranged saying "preserve layout in this folder" (which you only have to click the first time, of course it remembers along with the layout). (Also, make it more convenient to default all folders/finders to list-with-details view, for people who have any idea what they're doing.)

3. When the user opens Spotlight, have a button in the sidebar "index this drive for Spotlight" for any un-indexed drives present.

In all these cases, it's still super duper easy to use all these things, without always bothering people who don't want these things.

skore4• 11 years ago

then have a button
Have a button
have a button

That's your problem right there.

PinguTS2• 11 years ago

That is this kind of thinking of engineers vs. regular user.

You know why Windows users are so prone to viruses and such? Because they learned to click "OK". They learned it, because they always got the question/offer if they want to proceed or not. But a regular user don't understand those technical details and you know, they don't _want_ to understand those details. So we learned them to click "OK".

The best UX for the regular user is any UI where regular users have less choices of technical details.

I said that already the other day: as an engineer myself, I were chatting with another engineer about a product they have done (device integrated into a car for special purposes). I said to him, that something feels wrong with the interface. Yes, all the required information is present. Yes, probably I myself would have designed the UI the same way. But over the time I've got some sense in terms of UX, which a product makes it compelling or just doing its job.

The same you see with smart home. For how long there are HVAC controls out in the field by Honeywell and others? The better one are all programmable in all regards. But only Nest, which removed a lot from the interface made the big wave.

blueskin_2• 11 years ago

Well said. This is good usability.

moe• 11 years ago

No need to contrive lame excuses.

It's just bad software design, plain and simple.

rakoo• 11 years ago

I believe the way you describe is the better way for users, but the problem seems to be about the .files littering the whole filesystem. Why not have a top-level .hidden folder, where everything is put in, for any OS ? You'd have the OSX's .DS_Store, Windows' Thumbs.db, and whatever Linux-specific dot-file you can have.

I kind of see a parallel with applications willing to put their stuff in ~/.myappconfig instead of the central ~/.config/myappconfig, and it bothers me as well.

lultimouomo3• 11 years ago

Thumbs.db is a random proprietary format

I may be wrong here, but it doesn't look to me that .fseventsd, .Trashes and .spotlight are open, standardized multi-platform formats.

Unless random proprietary format means "My mac doesn't read it".

__david__4• 11 years ago

".Trashes" is just a directory. There's no "format" to speak of…

Your point stands for the other files, though.

Nevor• 11 years ago

Just a side note, Thumbs.db are no longer created in visited folders since Vista. Mac OS seems to be the last OS that randomly trashes removable mass storage.

frik• 11 years ago

First thumbs.db was a WinXP only thing. Windows Vista and higher store the thumbnails on C:\ in the users folder!

But actually the thumbs.db file was great! The thumbnails were local next to the files. With WinNT 6+ the thumbnail store grows several GBs on C: for each user. If one moves a directory it has to reindex it. Attaching a USB removeable drive increases your central thumbnail file on C: even further. There is a pruning feature so if you don't access the USB storage the data is overwritten. Then the Windows Shell has to index Terabytes of external data again. Win7 has a bug that every now and then updating the central thumbnail store (and using Windows Security Essential) consumes more memory (16+ GB) than is available - it was clearly never tested with 100.000s of media files.

It doesn't help that Windows Explorer starting with Vista lost several convenient features like remember the file list style of every folder or using a "folder.jpg" file as folder preview picture instead of showing 4 random pictures. Win8.1/10 at least brought back the "up" button. WinXP had also a real toolbar that could be changed, now you need a custom shell extension.

RexRollman• 11 years ago

Am I the only one who doesn't like file preview icons in my file manager? I always turn it off on Windows and OS X.

userbinator• 11 years ago

Hopefully users will push back; demanding to be given more options, and more control. We've gotten to a point where the role technology plays in our lives is too integral to keep letting that control slip away, one "harmless" feature at a time.

Amen to that. Unfortunately it seems the trend is towards hiding the filesystem/concept of files completely from the user and replacing it with app-specific isolated storage in a proprietary format, which is even worse than unsolicited filesystem modifications. (On the other hand, it could be argued that these modifications would become even less noticeable in such a system... which is good or bad depending on who you ask. I'm firmly on the latter side.)

A physical write-protect is something all removable storage should have.

piersadrian• 11 years ago

for the love of christ, it's OS X, not OS/X

jordigh3• 11 years ago

To distinguish it from its predecessor, Mac OS IX?

Trolling is a art.

yellowapple• 11 years ago

I want to see someone come out with an OS/2 skin of sorts for OS X, just to make everyone in the world that much more confused.

delinka• 11 years ago

And I hope you're pronouncing that as "Oh Ess Ten" and not "Oh Ess Ecks" ;-)

angelortega• 11 years ago

So that .Spotlight-V100 store some kind of index, probably binary. I presume that any Mac reads it on any arbitrary USB drive it's plugged in (or, at least, when a search is triggered). By creating a specially-crafted, corrupted index I see a channel for denial of service attacks.

dchest• 11 years ago

You can probably still abuse NSDictionary, causing HashDoS (https://gist.github.com/dchest/4467707). I did it with QuickLook a few years ago; don't know if they replaced a hash function there.

0x0• 11 years ago

Before OSX 10.10.2, apparently there was a "heartbleed" style info leak in spotlight where it "may save unexpected information to an external hard drive", according to http://support.apple.com/en-us/HT204244 (CVE-2014-8832). Probably related to this report: https://www.f-secure.com/weblog/archives/00002752.html

nfoz2• 11 years ago

It's just nonsense that this metadata would be dropped as files scattered across the filesystem, rather than simply held in a single location on the host machine. They're a fragment of the OS, not of each directory.

PinguTS2• 11 years ago

Which would mean that those meta data is only available on this host and not anywhere.

The problem is the short coming of those file systems not to allow to store arbitrary meta data to any file.

nfoz2• 11 years ago

That's exactly what I want though. "Folder display preferences" are a thing that pertain to exactly one instance of one operating system. If I want to share "folder displays" to a different computer (which seems rather unnecessary), that seems like an explicit action I might take between the two computers, and all that metadata should be a single bundled file that is transferred between machines. Not per-directory metadata that is bundled into my filesystem everywhere.

Doctor_Fegg• 11 years ago

I'd be more inclined to berate Apple for this were it not for the amount of crap that Unix-style tools dump in my home directory.

I count 50 .-prefixed files or directories at present, some of which is from Mac developers who should know better (hello, Adobe, Dropbox), some is cryptically-named gunk from other tools (.vegas, .jmf), some various command line histories (bash, fcsh, mysql, psql, pry, sqlite, irb), and so on.

OS X is nicer than traditional Unix behaviour here - it puts it all in a well-organised ~/Library directory.

rat87• 11 years ago

On Linux things should go in xdg dirs ~/.config ~/.local and ~/.cache instead of ~ but a lot of old unix software and sadly a decent amount of new software(even guis) don't follow the standard.

__david__4• 11 years ago

Even 50 is nothing. On my Linux machine:

    $ ls -d  ~/.* | wc -l
    342

kedean• 11 years ago

I got the impression that he's not upset about all the dotfiles, he's upset that they put them on external media. Most unix programs will only create those in the users home, which is usually not external.

nathanstaines• 11 years ago

Eject for Windows (http://www011.upp.so-net.ne.jp/decafish/EjectForWindows/Ejec...)

wampus• 11 years ago

What happens when the drive is shared among Macs? Why isn't this a vector for exploitation?

__david__4• 11 years ago

Well, if you find a buffer overflow that can lead to code execution in spotlight (mds) or fseventsd then it very well could be a vector. Assuming no bugs in those (hah), then I'm not sure what you could exploit…

chiph• 11 years ago

It seems to me, that if I insert a removable media that is formatted with one of the FAT filesystem variants, that I probably don't want those hidden files & folders created. As it's a sign that the ultimate destination probably isn't another Mac.

blueskin_2• 11 years ago

Never understood why they do this. Surely they should keep junk off removable drives, as they are the most likely to be used in other systems including ones that don't hide data from the user.

.Trashes, .fseventsd, and .Spotlight-V100

Related Stories