Rob Pike: the origin of dotfiles

https://plus.google.com/101960720994009339267/posts/R58WgWwN9jp

by keyist • 14 years ago

470 153 14 years ago

plus.google.com

keyist2• 14 years ago

Instead of putting a dotfile or dotdir in the user's home directory, do follow the XDG Base Directory specification: http://standards.freedesktop.org/basedir-spec/basedir-spec-l... .

It's easy to understand and requires only a marginal increase in effort/code.

oofabz• 14 years ago

I am always pleasantly surprised when programs follow XDG. One notable example is the fish shell. Most shells put dotfiles in $HOME, like .bash_profile or .zshrc. Fish puts everything in .config/fish/.

moe5• 14 years ago

I see no benefit.

It's not at all easy to understand, nor easy to implement, nor does it have any tangible advantages.

It's one of these superfluous pseudo-standards that do nothing but add needless clutter. But gladly nobody seems to be using it anyway, I see only one directory in my ~/.local: vlc.

subway• 14 years ago

Check out your .config and .cache directories.

keyist2• 14 years ago

I consider it good practice to allow users to customize where your program creates files. I don't like $HOME to be the default dumping ground for anything I run.

A standard that uses environment variables means programs don't have to provide extra options for this customization (I've seen -f,--file , -c,--config and other variants). It allows for common code (libraries that implement the spec).

If you poke around for feature requests for various open source programs, you'll find XDG basedir compliance come up occasionally (moreso for CLI utils). I wouldn't say "nobody seems to be using it"; quick scan of my folders includes Chromium, uzbl, htop. Git's next release will be compliant too.

sciurus• 14 years ago

~/.local isn't primarily where you should look; check ~/.config

darkarmani• 14 years ago

It's only used by those few users that use SSH.

aroman• 14 years ago

Actually, it is in considerably wide use for desktop linux applications.

It definitely has tangible benefits as well -- it promotes the clear separation of app data and user configuration, and unclutters the home folder.

And as pointed out by other replies, ~/.local is not the only XDG data dir.

I'm curious, how do you find it not easy to implement?

davvid• 14 years ago

I see no benefit.

How about consistency?

dfc2• 14 years ago

You are running a modern linux distro with only one file/dir in .local?

darkestkhan• 14 years ago

Interesting... I have ~/.cache, ~/.local and ~/.config and all 3 of them have quite a lot of subdirs and dotfiles inside.

gioele• 14 years ago

> I see no benefit.

The first benefit is that it removes clutters from your $HOME.

The second benefit is that you can now manage and backup your settings in a sane way.

* ~/.config contains config files (should not be lost, but if lost you can recreate them);

* ~/.local contains user data files (save them often, never lose them for they are not replaceable);

* ~/.cache contains cached information (can be tmpfs mounted, you can delete it any time you want, no loss in functionality, you just lose some optimization);

* ~/.run is for temporary system files (must be tmpfs mounted or it must be cleaned during at shutdown or power on).

Luckily most of the apps used on Linux systems now use it, you are probably using Mac OS X.

vrotaru• 14 years ago

    find .local/  | wc -l
    618

So I guess it depends on your Linux flavor. Mine is Ubuntu 12.04.

malkia3• 14 years ago

Somehow .dotfiles work just fine under Windows :). As such they are somehow more "portable"

Emacs saves them under %USERPROFILE% - I have in there - .alice, .android, .easyhg, .eclipse, .gstreamer-0.10, .lighttable, .m2, .matplotlib, .... .VirtualBox, .zenmap

Also in "Application Data" - .emacs.d, .mc, .subversion

My point is - this system works somehow even under non-unixy systems.

Because it's simple.

Dylan168072• 14 years ago

%APPDATA% is as simple as home but much better on the clutter front.

strictfp2• 14 years ago

It didn't use to work. I think it was added for XP SP2 or so, probably due to the popularity of manu *nix ports like Emacs.

jasomill• 14 years ago

Incidentally, Emacs will use %UserProfile%\.emacs.d if you set %HOME% to %UserProfile%, which, as a UNIX user, I personally prefer for consistency. While not Emacs-specific, another trick I've found useful is that, on 64-bit Windows, you can open %SystemRoot%\System32\somefile in 32-bit Emacs iff you refer to it as %SystemRoot%\sysnative\somefile (the original path yields %SystemRoot%\SysWOW64\somefile).

joe_the_user2• 14 years ago

I notice the $XDG_DATA_HOME environment variable isn't set in my Intrepid install. Maybe its set in later Ubuntu version.

But the value not being set for me means implies to me that it not being set is pretty common. Thus it seems like supporting this standard is going to involve supporting a fall-back of whatever you would do otherwise.

So I don't see "easy" at all but rather extra BS. Sorry.

angusgr• 14 years ago

From the spec: If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used. Similarly there's a default (.config/) defined for XDG_CONFIG_HOME. So the fallback is well-defined rather than "whatever you would do otherwise".

Also, for my 2c, you should consider updating your Intrepid install if you at all can. It hasn't been supported for over two years, so it hasn't seen any security updates in that time. The Ubuntu do-release-upgrade system is pretty easy and reliable.

jeltz• 14 years ago

I would have preferred if it did not rely on environment variables since almost nobody sets them which means there will be untested code in applications implementing the specification.

takluyver• 14 years ago

Where possible, applications should rely on tested libraries to implement the specification, like PyXDG.

yaix2• 14 years ago

On my system (Xubuntu), these environment variables are empty. Seems not to be much of a standard.

saljam3• 14 years ago

To be fair, that's how it works. They have default values when they're not set. These are for when you want to override them, e.g. to keep cache files local instead of over your net mounted home directory.

tyrmored• 14 years ago

This is a problem that libetc is supposed to help solve: http://ordiluc.net/fs/libetc/

malkia3• 14 years ago

It also begs the question, why it was named "etc" :)

napum• 14 years ago

I'm guessing it's because /etc contains configuration files and dot-files are just configuration files. Or do you mean, why was /etc named etc?

spitfire• 14 years ago

You can ask this question for most of unix. Why /etc? Why /bin and /usr/bin? (Answer: At one time hard disks were very small and crashed a lot), why do we presume screens are black and white, etc, etc.

Try to change any of it though, and a lot of luddites will come out screaming bloody murder. It's just not UNIX if it makes sense.

noibl• 14 years ago

Maybe the alternative was "..." :)

szopa• 14 years ago

The origins of /etc are lost in history. Wikipedia [1] says that at Bell Labs /etc was pronounced "et caetera," and contained files that didn't belong elsewhere. And it had the advantage over conf or misc that it was only 3 letters.

[1] http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

agumonkey2• 14 years ago

my de-obfuscation attempt:

etc > e.t.c > edit to configure.

the .rc suffix has a nice history btw

deno2• 14 years ago

That’s nice, shame it’s not maintained. On the other hand, one the goals of XDG is to differentiate between application cache, actual settings etc. If FooApp stores 2GB cache file in .foo, redirecting it blindly to .config just makes my backups that much harder.

skeletonjelly2• 14 years ago

If not dot files lazy programmers would have just found another way. Just look at the results of lazy programmers on any Android SD card.

voltagex_• 14 years ago

Is there a preferred standard for using the SD/external storage on Android?

skeletonjelly2• 14 years ago

http://www.reddit.com/r/Android/comments/t3x0p/randroid_lets...

> I do find it odd that so many apps have this problem. It's trivially easy to get the proper location. Just call getExternalFilesDir() . Deletion at uninstall happens automatically. In fact, it's the ONLY way to make sure those files are deleted when you uninstall, because you can't run code at uninstallation.

chj• 14 years ago

Yay, I am one of them.

seanalltogether2• 14 years ago

For those who object that dot files serve a purpose, I don't dispute that but counter that it's the files that serve the purpose, not the convention for their names.

I would like to hear a good argument for why hidden files and folders are a good thing.

frossie• 14 years ago

They are a natural segregator of novice and advanced users. If you have a limited level of interaction with the unix shell (remember than in the old days everyone in a science academic department used a Unix machine, even the dusty professors you kept in the back of the supplies cabinet), setting up their account and then making sure that they couldnt get into any trouble because they didn't know about 'ls -a' was very much a feature, trust me.

Moreover I personally much prefer them to global configuration directories because they are always local to the thing being configured and you can always override a global configuration by using them.

In fact I think they are a very elegant way to handle "hidden options" - stuff you want to expose to the power users but not bother newbies with.

tl;dr: I am not convinced they are a misfeature.

seanalltogether2• 14 years ago

I wonder if that is a good solution for separating novices and pros though? For me, hidden files and folders only help to reinforce the confusion people have about computers. It's one thing to provide a folder to a user with a lock or shield on it, it's another to completely hide it from them.

SoftwareMaven• 14 years ago

OS X has a ~/Library directory that houses application settings and much else. It is hidden by default.

There is no question there is a need to have a place that "plebeian users" can't access. IMO, dot-files and dot-directories are as good as anywhere.

joe_the_user2• 14 years ago

Lol,

They are a natural segregator of novice and advanced users.

Yeah, they let that guy in the computer lab who "really knows Unix" show his stuff.

I remember encountering these dot-whatever files back in the day, how changing all the idiotic terminal settings depended on them and how remembering their names or interpreting their values was nearly impossible, and how the cool geeks of the lab had about six seconds of their time available to explain the situation.

bdunbar3• 14 years ago

I would like to hear a good argument for why hidden files and folders are a good thing.

They keep users from monkeying with stuff until they are smart enough to find the hidden files.

Xcelerate2• 14 years ago

*knowledgable enough

tikhonj• 14 years ago

They work well with per-directory settings. It's a simple convention and it works really well for things like .git, .gitignore, language-specific settings files and so on.

Putting files and directories like this inside another directory would not win you much, especially if you only have one in a given location (e.g. you only have a .git directory). But you wouldn't want to keep tripping over your .git directories either!

However, having a dedicated directory for configuration files applying to a user or the whole system is a different story. I would much rather just have a ~/config directory than a home folder full of dot files.

tedunangst3• 14 years ago

I type ls in my home directory quite frequently. Somehow stuff accumulates there and I need to find a better place for it. I never type ls -a, precisely because there are a million files I just don't care about. Some means of saying "this file is here if you want it, but it won't show up in a listing because it's unlikely to be the file you're looking for" is convenient.

dsrguru• 14 years ago

That's really interesting. I always assumed hiding dotfiles was a deliberate convention, but to semi-quote one of the commenters, Rob's got a point. Or two.

adambratt• 14 years ago

Heh, I remember when I first found out about dotfiles. I thought it was a genius idea...

Little did I know.

emperorcezar2• 14 years ago

If someone thinks hidden files are a misstep, then I don't want them designing a OS. Grandma really doesn't care about some conf file, or anything like that. She cares about the pictures of her grandkids.

The opinion that there shouldn't be hidden files comes from a perspective of someone who is a "power user" and who can't step back and see that most users really don't care for some .config file. To them it's clutter that gets in the way of finding what they really care about.

That said, dot files maybe the wrong way to do it. I like ~/Library in OSX. That's one good way.

Edit: Note that I'm talking about a general trend in the post's comments and on here. Not the author's opinion.

UK-AL• 14 years ago

Did you read his post? He said exactly that. Put them in a config directory of some sort.

emperorcezar2• 14 years ago

Yes. I'm not talking about his post. I'm talking about a few other comments here and on G+.

SCdF• 14 years ago

Sounds like you agree with him, as he advocates a 'conf' directory at the end of his post.

rehack• 14 years ago

I think the case of hidden dot files is a good example of 'convention over configuration'[1].

Point is accepted, that it came into being due to a lazy programmer. But surely early people might have just liked the unintended consequence of some files (dot files) being hidden. Just like most of us, whenever we learnt unix thought that it is by design.

If early users, had found the consequence a handicap, it would have been fixed long back.

Its similar to the use of hash-tags on twitter or the @for addressing which got adopted by users first and so became features (although the paths to them being considered features are different).

[1] http://en.wikipedia.org/wiki/Convention_over_configuration

Edit: Grammar

ilaksh• 14 years ago

The main takeaway I think is that the reason we are doing things the way we do them is because that is how we have been doing them, not because of some kind of genius design. And actually the whole thing can start out as a mistake, and then later on basically becomes a religion. Its very funny actually.

mturmon• 14 years ago

Yes, very organic. The original programming shortcut accidentally created a sort of ecological niche, in which all sorts of things started to take root. Now, nobody can bulldoze the lot because there's too much stuff growing there.

eliben• 14 years ago

I really like it that G+ is becoming a "lightweight blogging" platform. There are too many of these around, and folding them into a "social network" seems like a good idea. I wish they would add more features that would make this easier, though, since in general I think it's in everyone's interest and will pull more traffic to the site.

89a2• 14 years ago

I'd like it if I didn't need a Google account to read anything on it.

jkn• 14 years ago

Well be happy then, you don't need a Google account, as can be seen by entering the url in an Incognito window.

djeikyb• 14 years ago

Yeah, it's pretty lame that if I'm signed into Google but don't want to accept G+'s real name policy, I can't read otherwise public posts.

lmm2• 14 years ago

Adding a new one is not a good solution to "there are too many around". And I don't think consolidating everything into google is a good idea; these kind of posts belong on a different network to photos of my holidays.

saljam3• 14 years ago

A bit off topic, but is it just me or is the quality of comments on public G+ posts is absolutely terrible? Within a few hours there has been more than 100 replies. Most are inane, rude, of the useless “great post!” type, or just plain old spam. I could only manage to glean about 8 or 9 insightful replies.

Is this something intrinsic to G+? Is it a function of the author's popularity? The discussions here are normally much more sensible, but I would expect HN's readership to have a fairly similar demographic to that of Rob Pike's G+ subscribers.

jff• 14 years ago

That was exactly my thought--there were a few thoughtful replies in there, but otherwise just absolute mindless bullshit. It's the same way when Linus makes a post about how he's just updated the code in his preferred emacs editor, spawning 500 posts of "I use Vim!"

dredmorbius• 14 years ago

G+ is fair to middlin' at a bunch of stuff. Comments lack threading. There's poor noise control. Content filtering is limited to "+1" or "flag". There's no "-1" button (though various Chrome extensions have offered this at various points.

The main advantage is that it's a large community (10m+ users presently) initially seeded by Googlers (e.g.: tech-savvy people), and including a few notable luminaries such as Rob Pike. And if you've got the right circles, eventually good content finds its way into your stream. Sometimes (content discover/surfacing is something G+ does surprisingly poorly, and is an area at whch HN, Reddit, and StackExchange win hugely).

caf5• 14 years ago

The fact that "." and ".." were allocated actual directory entries and were returned when reading the directory, rather than just being handled by the kernel when parsing pathnames seems like the original sin of expediency here.

jlgreco5• 14 years ago

Why? Seems like a brilliantly simple quick solution that would be rather easy to roll back in the future and lacks any real downsides besides being a tad weird.

caf5• 14 years ago

Because it was creating real directory entries that caused "." and ".." to be visible to userspace programs reading a directory, which then led to the hack in "ls" to hide them, which is where the article picks up.

It also isn't that easy to roll back once userspace programs start to rely on it - for example, the assumption that the number of files in a directory is equal to st_nlink - 2 is now so widespread that it's part of the UNIX API.

VMG• 14 years ago

Plus you can find out the directory inode and the parent directory inode

mseepgood• 14 years ago

He also writes why the dd command is so horrible: "dd is horrible on purpose. It's a joke about OS/360 JCL. But today it's an internationally standardized joke. I guess that says it all."

http://en.wikipedia.org/wiki/Job_Control_Language

acabal2• 14 years ago

I hate having dotfiles in ~/ for the same reason why I hate "My Documents" in Windows: because it's supposed to be my space that I organize, not a generic dumping ground for your config files, brand-named folders, or other nonessential garbage.

I want my space to be mine. Keep your app's stuff out of there!

chris_j• 14 years ago

Where would you choose to put application config instead?

acabal2• 14 years ago

Literally anywhere but my personal space. Why not have /etc/<username>/? I could even live with a single ~/.config/ but very few apps use that.

Foy• 14 years ago

On Windows I've retreated to Desktop which remains fairly sane as long as I opt-out or remove shortcuts being created after an install.

"My Documents" is a nightmare... I think only one folder (out of nearly a dozen) and only one file are actually of my making.

zapu• 14 years ago

Microsoft now encourages to use AppData/ which is much better.

gnufs• 14 years ago

I am using a manually created /data partition as my personal space. Many dotfiles and 'dotdirectories' are distro+release specific. With having all my personal data in a separate partition makes switching distros (or using multiple ones) a breeze as I start with all my data intact and a virgin /home which I can populate with my dotfiles as I see fit.

mavroprovato• 14 years ago

My Documents was never meant to be a storage space for application settings. Application Data / AppData is the place for this. Of course, whether an application complies with this depends on the developer, and many applications break that rule

rm445• 14 years ago

But My Documents is one level below your homedir.

Your homedir equivalent on windows (since Windws XP) is /Documents and Settings/username, or the modern shortened equivalent. That has subdirectories for documents and application settings. Apps that put config data in My Documents are using behaviour dating back to Windows 95.

It's the same on Mac OS - /Users/username has a Documents subdirectory. Mac spps tend to be better-behaved, but a few manage to pollute the Documents directory. On Linux you may roll your own or use a ~/Documents directory created by your desktop environment. The fact is your homedir is the home of pretty much anything that is user-specific, and you either need to collect all the config data into some sort of config directory, or do your own organising in a subdirectory, or both.

garfee• 14 years ago

omg!

89a2• 14 years ago

Can't read without signing into Google…

oh well nothing of value was lost.

mproud• 14 years ago

I'm with you.

antirez• 14 years ago

dotfiles are not perfect, but to have this very negative vision on a feature that also helped is a bit a revisionist attempt IMHO.

Dotfiles provided a poor, but at least simple way to store program-specific-user-specific configuration, since another standard was missing. After all it's a simple and decentralized system that worked very well with the concept of unix user and ACL: you write something inside your home directory, and this changes the behavior of your program.

Consider that this was invented many decades ago. Now it seems a lot better to have directories with sub directories. Maybe back then it was considered to be a waste of resources, inodes, and so forth.

We can improve it, create a new standard, and have something better than dot files, but dot files are better than many other over-engineered solutions that I can imagine coming out of some kind of design commission to substitute them.

Every time to passed your vim configuration to a friend you just copied a text file, sending it via email: you enjoyed one of the good points about dot files. Every time you did something like cat dotfile | grep option you enjoyed the positive effects of single-file plaintext configuration.

Also it's worth saying that dot files are not just the concept of an hidden file with config inside. A lot of dot files also have a common simple format of multiple lines "<option> <value>", that's better than some XML or other hard to type format (IMHO JSON itself is not good for humans).

ajanuary• 14 years ago

How does any of those advantages apply only to dotfiles and not just plain files?

FuzzyDunlop• 14 years ago

I think there's an aspect of convenience, too. Personally, I prefer that to, say, configuration stored in some unspecified location in ~/Library/Application Support/Application/* (on OS X).

leothekim• 14 years ago

"How many bugs and wasted CPU cycles and instances of human frustration (not to mention bad design) have resulted from that one small shortcut about 40 years ago?"

Sigh, if only most of us had worked on a software system that has lasted as long as that.

specialist• 14 years ago

Tangent: I'm grumpy that Eclipse IDE uses the file names .project and .classpath. So they're hidden by default. Requiring special treatment.

Their content is XML. What's wrong with project.xml and classpath.xml?

mcgwiz• 14 years ago

The author's gripe seems to be that the hiding of dotfiles was unintended, ergo dotfiles are Bad. Whether they were intended is irrelevant; their wide usage vindicates the practice. After all, traction = value. The problem of program state/configuration/metadata storage is adequately met by dotfiles.

There are, no doubt, numerous unintended behaviors of programs. Most of these are simply ignored and certainly not leveraged the way the dot behavior is.

People don't go out of their way to abuse an unintended system behavior; they simply leverage all capabilities of a system ("intended" or not) to meet their needs. Had dotfiles not gained traction, some other solution would have been designed (or "engineered") to meet the needs of program state/configuration/metadata storage.

[Tangent: All of this reminds me of grammar freaks that harp on "correct" usage, completely oblivious to the fact that grammar changes, and "correct" is merely a lightweight pointer to the current norm.]

Rob Pike: the origin of dotfiles

Related Stories