TMSU: a tool born out of frustration with the hierarchical nature of filesystems

http://tmsu.org/

by goblin89 • 10 years ago

237 128 10 years ago

tmsu.org

x5n13• 10 years ago

It seems to me most of this stuff can be done via unix command line if you are so adept... without this app.

zyxley• 10 years ago

What existing tools can automatically generate a virtual filesystem based on tag metadata?

x5n13• 10 years ago

no not virtual filesystem, i meant querying your files by date or tag... find, grep and something to print the tag would do the job.

cyphar• 10 years ago

Tagging can be done with symlinks if you just want to stick to using the filesystem.

marklgr2• 10 years ago

You can have some poor's man tag management with subdir-per-tag containing softlinks, but it gets unwieldly pretty quick.

tyingq• 10 years ago

It's not terribly portable, but you can use extended file attributes.

Here's an article showing how they work for Linux, OSX, and FreeBSD: http://www.lesbonscomptes.com/pages/extattrs.html

zaphar6• 10 years ago

I'm actually not frustrated with the hierarchical nature of filesystems. I'm most frustrated with the state of filesystem search these days. I don't want to tag and curate my files. I want to search them.

I've strung together something using bleve full-text search and some OCR libs to scratch my particular itch but it still doesn't quite get all there.

kpil• 10 years ago

I agree.

I actually like the hierarchical organisation, and I don't like the 10 year usability trend, particularly driven by Microsofts attempts to patch over their horrid structure with even worse workarounds.

The solution to learning my mother were to find her documents is not hiding the place 15 levels deep and having 10 symlinks to it.

Fast, unobtrusive indexing and a good structure is all that is needed. Humans organise and memorise things in hierarchies, and while categories is another helpful abstraction they tend to be confusing if they do not point to a fixed position in a hierarchy in my opinion.

You can also explore a hierarchy; Not so with a cloud of things connected to a bunch of tags.

ori_b2• 10 years ago

Tags are a subset of hierarchies + symlinks. If you have a one level hierarchy, you have the equivalent of tags.

vlunkr• 10 years ago

You have to realize though that this software doesn't take away your heiracichal system, it just gives you another way to look at it. Recently I've been cleaning up my music library and I'm seeing some of the shortcomings of the current system. What do you do with a compilation album? Have a folder for each artist or put them all together, (Which would separate them from others works by the same artist)? iTunes or whatever media player you use will probably list them in both locations depending on how you sort or whatever. I find this to be a useful feature, why wouldn't you like to have the option to do that sort of thing from the command line?

igravious4• 10 years ago

Humans organise and memorise things in hierarchies

Big claim. Proof?

blakeyrat3• 10 years ago

Humans organise and memorise things in hierarchies,

Humans also track a lot of things based on spatial memory, which no OS since Mac Classic has even tried to make use of, which is a shame.

choosername• 10 years ago

Humans organise and memorise things in hierarchies, ...

Well, sure, if things do have a hierarchical relationship, but a hierarchy is just one form of network. It fails for concurrent dependency problems, e.g.

qwertyuiop9242• 10 years ago

Yeah. I just want my files in directories, save maybe music, where I might want to query by various tags. But beets already can do that. Or I suppose I could build an app to build a static index off of a master, but again, why would I do that when I can use beets?

All of the other files I work with on a day to day basis I don't really want to query by structure or tags. I just want to know where the %^&*@ they ARE! And a simple hierarchy is actually better than tags for that.

kujenga• 10 years ago

I hear you there! If you're looking for a tool that scratches that itch, I've spent the last year and a half working on a could based search engine that spans all services and devices.

We inherit all the information from your existing file system structure, as well as the people files are shared with, and provide a powerful content-based search on top of that.

Check out our website if you'd like: https://www.meta.sc

zaphar6• 10 years ago

If my files include a scanned in PDF with text that isn't stored as text per se will meta.sc index those? My homegrown solution does this for the most part and that feature was the motivation for creating it.

That and the fact that it's available offline on my machine.

tremon4• 10 years ago

I'm not a big fan of semantic search itself, nor do I enjoy meticulously tagging my files. But I would love to have different views of my files, based on different metadata.

I know that I'm not always searching for files along the same hierarchy: sometimes I want to search along a timeline, because I know that I edited two files around the same time. Or I remember where I was when I wrote something, and I'd like to search on geolocation. Sometimes I know that I've implemented a similar feature for a different project, and want to copy/review what I did then.

All these searches still have hierarchical components in them: in the "date" case, I would expect to browse to a generic "N weeks ago" directory, look at a few specific items to get my bearings, and start browsing deeper to narrow my date range, or move forward or backward in time. In the geolocation case, I'd want to pull up a map, click on a country, then a city or area, and look at the files I accessed while there. As for the project example, I probably have the project files lying around in a hierarchical directory already. But I'd want to access only the relevant feature file, not browse the entire project structure again.

For me, the problem with non-hierarchic interfaces isn't just the lack of metadata: it's a lack of tools for visualizing non-hierarchic matches. Every search I described above requires its own GUI...

zaphar6• 10 years ago

I don't really need semantic search. Full text search covers 99.9% of my needs.

copperx3• 10 years ago

I actually made a sibling post highlighting almost the same paint points as you, which makes me think we may be onto something ...

copperx3• 10 years ago

There needs to be a middle ground between hierarchical organization and search. Hierarchical organization is painful because an object can only be in one place in the hierarchy (disregarding links, which are a hack), and search by itself (as implemented in e.g., Spotlight) is not optimal because spatial/temporal/conceptual clustering of documents is important for discovery of related materials.

I have more than 5 todo.txts scattered in my system. I have cloned my webpage's git repository countless times in my local drive. I have countless LaTeX documents named letter.tex.

Of course, if I search for letter.txt I get a hit, but what other documents in my system did I create around the same date? were any other associated file changes? these questions are hard to answer in a hierarchical filesystem.

While we're at it, we should separate the task of document naming and classification from document saving. Right now, if I create a document in say, Microsoft Word and I want to make sure it gets saved, I have to pick not only a discoverable filename but also the right place for it in the hierarchical tree. I am just trying to type up a quick letter and get it printed, not grapple with the philosophical question of "where does this file belong and what should I name it?"

noja• 10 years ago

Why don't you put everything in one folder and name it with what the document is?

deprave2• 10 years ago

UI nitpicking:

These:

$ tmsu tag summer.mp3 music big-jazz mp3

$ tmsu tag --tags "music mp3" foo.mp3 bar.mp3

$ tmsu tag spring.mp3 year=2003

Are confusing. Very non-Unix. They should be:

$ tmsu tag music,mp3,year=2003 summer.mp3

RubyPinch• 10 years ago

I have to ask, how are they non-unix?

deprave2• 10 years ago

They're inconsistent. Sometimes the file name comes before something and sometimes after something. It's a lot easier to remember commands if their structure is uniform, and Unix commands usually have the general structure of "program [options] [files]"

AceyMan• 10 years ago

I empathize with the OP's view that there's a usability gap with out of the box file systems/namespace (short of bash/sed/awk/perl wrangling, natch), so I applaud this project.

I have a couple of points/concerns after a quick read.

1) So 'tag' is the verb that updates the records, but 'tags' is a read?

Poor taxonomy, IMHO. If I'm using "tmsu" I know I'm working with tags, so I'd think natural switches would be the way to go ( tmsu add, tmsu ls|list, &.) Or at least don't make the "create" verb and "read' verb differ only by the plural 'S.'

2) Is there a way to list all existing tags? —not just the ones already bound to a file but all available in the database (with regex filtering, of course).

That's what I'd need to pick from my 'tag pallette' before actually tagging so I could avoid creating synonyms accidentally that'd later require a merge.

gumby• 10 years ago

Why not use file attributes and provide a nice interface for managing them? (e.g. extend find to search them, etc)? A parallel database is fragile -- it can trivially get out of synch.

codezero• 10 years ago

I thought the same thing, but in practical terms you'd need to maintain a separate db for portability anyways I guess.

crescentfresh• 10 years ago

So tag order is important?

i.e. `music mp3 folk` results in a different virtual file system than `folk music mp3`?

I often think of tags as an unordered set, rather than an ordered list.

GordonS• 10 years ago

An ordered list smells a bit like a hierarchy to me...

zepto• 10 years ago

Seems like the Tags feature that has been in MacOS X since 10.9

ghshephard• 10 years ago

Was just going to post this - you can also access them from the CLI with mdfind command, mdfind tag:jazz

dbm5• 10 years ago

https://github.com/jdberry/tag

edward• 10 years ago

You can do the same thing with git-annex: https://git-annex.branchable.com/tips/metadata_driven_views/

seagreen4• 10 years ago

A great idea! I think tags are definitely a better way to organize most personal data than trees.

Also I like that they describe what data they actually change on your computer right on the homepage: "TMSU does not alter your files in any way: they remain unchanged on disk, or on the network, wherever you put them. TMSU maintains its own database and you simply gain an additional view, which you can mount, based upon the tags you set up."

Unfortunately building on a foundation of sand (meaning not TMSU's code, but Unix filesystems) has downsides:

https://github.com/oniony/TMSU/wiki/FAQ#why-does-tmsu-not-de...

" Why does TMSU not detect file moves and renames?

To detect file moves/renames would require a daemon process watching the file system for changes and support from the file system for these events. As some file systems cannot provide these events (e.g. remote file systems) a universal solution cannot be offered. Such a function may be added later for those file systems that do provide file move/modification events but adding support for this to TMSU is not a priority at this time.

The current solution is to periodically use the repair command which will detect moved/renamed files and also update fingerprints for modified files. (The limitation of this is that files that are both moved/renamed and modified cannot be detected.) "

Ouch.

kbenson2• 10 years ago

Hmm, seems like they could have gone the other way, throw everything into a DB, and then wrote a fuse plugin to access it all through traditional file system mechanics. That would have allowed for gating direct access such that moves and renames could be dealt with accordingly. Of course, there are other problems with that approach, but probably not as many as you might think (the file system is a database, so you're really just choosing a back-end that is less likely to be directly accessed).

seagreen4• 10 years ago

    they could have gone the other way, throw everything into
    a DB, and then wrote a fuse plugin to access it all
    through traditional file system

This is the Camlistore strategy!

    Of course, there are other problems with that approach

Could you elaborate more on these? I've never worked with FUSE.

ChuckMcM• 10 years ago

ding ding ding! This is the monkey in the wrench as it were.

Tagging is a really useful idea, it is also a naming thing and as such either it lives in the naming infrastructure (aka dirents) or it rots over time. A simple example I used to use in the 'object naming' [1] days was, imagine that instead of house numbers on the street you wrote down last names. That works fine until somebody moves and now not only did you show up at the wrong house, you don't even have a chance of knowing what the correct house is. [2]

Microsoft's LongHorn project was way out there but took a swing at the actual problem. Just make the file system an actual relational database. Then your home directory is simply 'select * from files where (owner = chuck);' It really does solve the problem at a more fundamental level, using naming by attribute rather than mapping. I got to observe that effort from the outside (I was at NetApp at the time) but I believe it died due to really horrible performance issues.

I find it pretty awesome that people can lose files, back when a "big" hard drive was 100MB it really wasn't all that hard to just look through all the files on it, but when its a couple or three terabytes, all bets are off!

[1] Object File systems were all the rage in the early 2000's, files themselves were object ids and the naming was a database that connected object ids to user recognizable names. -- https://en.wikipedia.org/wiki/Object_storage

[2] The typical solution is to add "tombstones" or redirects at the previous address. That then is a layer of additional meta data to maintain, and sometimes the file doesn't move, it just changes value (trivial example you have a file 'my-favorite-song.mp3' which is tagged 'jazz mp3' and then you discover techno and make something from Tiesto your favorite song and while the name and type are still valid, the tag 'jazz' is now invalid.

GauntletWizard• 10 years ago

I've been considering writing something similar myself, and my plan had been to hardlink the files by their hash into my blobstore. It won't fix the move/modification case, but it would solve the problem for simple moves. But I guess they're trying to deal with remote filesystems, too, and I was not targeting those.

cvwright• 10 years ago

Couldn't they just create a hardlink in a private, hidden directory that they control, and then symlink to that?

Then, it's OK if the original file gets renamed or moved, as long as it stays on the same FS. You still have your hardlink, and so your symlink still works.

ZenPsycho• 10 years ago

what if you really want to delete the file tho? (passwords, customer data, incriminating evidence) then you have to remember to delete it from this system too!

Chris2048• 10 years ago

It would seems to me a better idea to order the VFS around queries - a command-line command returns a query id, which represents a directory in the VFS (such that the command is a bit like a mkdir for the VFS), the dir might then be '/foo/tmsumountpoint/<queryid>/', and contains symlinks for all files found in the query.

I'm sure FUSE can do this.

DannyBee• 10 years ago

This reminds me of BeOS's BFS, which let you do all this as part of the FS itself :)

swalsh• 10 years ago

I seem to recall reading somewhere that one of the reasons Vista was a "bad" os, was that it originally had much higher more optimistic ambitions to build a sql like file system (i suppose similar to this). It however had issues, and a decision to scrap it delayed the vista project, and reduced the scope of "cool" things it was supposed to deliver.

teraflop• 10 years ago

Yep: https://en.wikipedia.org/wiki/WinFS

jandrese3• 10 years ago

An interesting chart appears on the Wiki explaining how Microsoft keeps tilting at this windmill and has two decades of cancelled or scaled back products to show for it.

I suspect there is an inherent scaling problem with the approach. While it demos well and can be useful in a lab setting, the full scale version ends up being so complex that mere mortals aren't able to comprehend it and end up just losing their files constantly in the morass of schemas and structured accesses.

The talk on the wiki about being able to write an editor that can handle any file type smells of pure insanity to me. An application that can present a usable editing interface for any filetype would be so bloated that it's almost inconceivable to think about a single company writing it. Can you imagine the kind of application that has an interface for manipulating photos, writing reports, tabulating data, programming (all languages), doing CAD, editing textures, composing music, etc...?

rincebrain• 10 years ago

The thing you're thinking of was indeed supposed to be in Vista, then post-Vista, and then scrapped all together - WinFS.

https://en.wikipedia.org/wiki/WinFS

marcosdumay• 10 years ago

It was promised for Cairo (that become win95), under the name of Object File System.

Different name, exact same promises.

leanderleeco3• 10 years ago

Hey guys, you should look at diamond.io, we're building something that is a tool to help you solve this problem and the problem of organizing information in general. Check us out!

seagreen4• 10 years ago

"It's backed by a powerful Artificial Intelligence."

Ahhhhhhh!

You're tackling a laudable goal, but it would help to link to a page with more technical details.

leanderleeco3• 10 years ago

Haha, I don't want to go too much off topic of this thread, but essentially we can analyze actual file contents, source and metadata to find patterns between file structures and associate them to user and "common" labels.

On top of that we try to get further accuracy by using user information to help us find the context. We call it a Personal System, a repo of knowledge and learned preferences heavily tailored around each individual user. We're just in beta now but definitely put your email down if you want to try it out eventually!

auganov• 10 years ago

I just keep all my work-files (documents, downloads etc) in my desktop folder (it's the "top-level" folder in Windows when you Alt-Up, have it symlinked at /desktop/ too).

My default view is a detail view sorted by "date accessed" (descending) which is what I need 99% of the time. Especially handy when uploading random images, quick edits etc from the browser.

btw I highly recommend https://pathcopycopy.codeplex.com/ for those that use a terminal and win explorer at the same time a lot.

Skunkleton• 10 years ago

How is this different than something like this:

    mkdir -p ~/tags/{music,big-jazz,mp3}
    ln -s /path/to/summer.mp3 ~/tags/music/
    ln -s /path/to/summer.mp3 ~/tags/big-jazz/
    ln -s /path/to/summer.mp3 ~/tags/mp3/

From there, you can use all normal filesystem tools to interact with your 'tags'. You could extend this with a simple script that handles duplicate file names in the same tag by sticking a hash of the file before the extension. Having a separate database for this information seems unnecessary.

joshstrange• 10 years ago

* 4 commands vs 1

* Doesn't account for files with the same name

* No helper methods like merge when you typo

Just a couple of reasons off the top of my head

Udik• 10 years ago

It's still hierarchical. Try to tag a file both as jazz and as party music with folders. You can't.

Edit: Misread. Yes it would work. Well, consider this a tool for doing it automatically, without filling your hd with bogus folders and links

noonespecial• 10 years ago

This supplies you with only one relation between the tags: "And Then". Ex. 'Path' (and then) 'to' (and then) 'summer.mp3'. Or 'tags' (and then) 'big-jazz'.

What you want is to also have AND, OR and NOT available. How would you find 'music' (and) 'mp3' (and not) 'big-jazz'?

dsp1234• 10 years ago

I'm not saying this would be simple, or as nice as the tool given, but it's not impossible.

First, do a 'ls' directory dump of the 3 folders. Then 'cut' out the softlink destination from each file. This will end up with 3 files with soft links. From there, 'grep' the music and mp3 files together to get a list of soft links that are in both music and mp3. Then you can do an inverse 'grep' to remove softlinks that are in big jazz.

Even if the system creates hard links instead of softlinks, then the same could be done via inode numbers

jfb• 10 years ago

Think of the poor inodes!

Mister_Snuggles2• 10 years ago

Stuff like this is cool, but it really points out just how poor the filesystem is for organizing certain types of data. The "Why does TMSU not detect file moves and renames?" question in the FAQ really highlights how this isn't helped along by the filesystem at all.

One comment mentioned BFS, which had some really cool stuff. There's an Ars Technica article that touches on some of it[0].

The secret to BFS, in my mind, is that applications use it. The Haiku Mail app, as noted in the article, used the filesystem as its email database by attaching its own attributes to messages. This is also an example used in the "Practical Filesystem Design with the Be Filesystem" book[1].

Unless the metadata becomes a first class citizen in the filesystem, any attempts to layer it on top will have problems. Either applications won't understand it or normal filesystem operations will cause the metadata database to become de-synced with the filesystem data.

[0] http://arstechnica.com/information-technology/2010/06/the-be...

[1] http://www.letterp.com/~dbg/practical-file-system-design.pdf...

JadeNB2• 10 years ago

Unless the metadata becomes a first class citizen in the filesystem, any attempts to layer it on top will have problems.

Remember when it seemed like Mac OS might give us a modern era of rampant metadata (http://arstechnica.com/apple/2005/04/macosx-10-4/6)? Ah, those were the days.

Mister_Snuggles2• 10 years ago

Yup.

MacOS does an OK job of helping the user find things with Spotlight, but it's not a full metadata system like BFS had.

Mail.app, for example, keeps each message in a separate file[0] (and probably has a cache or separate database of this to make displaying mailboxes quicker). This makes it easy for Spotlight to index, but all of the stuff that you'd think of as metadata is actually just regular data inside the .emlx file.

If Apple made a huge effort to start treating the metadata (assuming the infrastructure described in the Ars Technica article still exists) as a first class citizen and using it like BeOS did, maybe we can get there. This would be a drastic rethink though. It feels like files, in some ways, are becoming second-class citizens in the Mac world. Photos, for example, are managed in the Photos app - you do not go into the filesystem and organize your photos.

One big problem with filesystem metadata is how do you transfer it? The Ars article showed a sidecar file (._filename) being created when the file was copied to a non-HFS volume. Now the metadata is detached from the file and we're back to the same problem.

[0] http://mike.laiosa.org/2009/03/01/emlx.html

Zikes• 10 years ago

Wasn't Windows 7 supposed to be built on a revolutionary new filesystem based on a rdbms? I was pretty excited about that because it would have natively enabled a lot of the features listed here. Unfortunately it was one of the features they cut when Win7 went over budget and over deadline, and they never brought it back for subsequent releases.

lucb1e• 10 years ago

I looked for a tagging system months ago, did fairly extensive research into existing solutions (rather than writing a FUSE layer myself) and TMSU was the leading result. I installed it and it's all ready to use.

Today, it's still all ready to use. I haven't touched it. I'm actually quite happy with the way my filesystem works, I just had this idea how great it would be to work with tag selections instead.

The only reason I might still use a tagging system is to tag some files I want to back up manually (if at all), like a 50GB disk image or some temporary big download, but in general I create one or two symlinks a year and I'm good. The hierarchy works fine.

stordoff• 10 years ago

in general I create one or two symlinks a year and I'm good. The hierarchy works fine.

Same here. I have a few things that tags would be nice for, but it's infrequent enough that current filesystems are fine. Couple of examples

* Tagging the source [CD/iTunes/Amazon etc.] of my music - tags would be nice (and possibly doable as IDTags etc.) but "/music/source/artist/album" or "/music/artist/album [source]" works fine

* Multiple paths to the same file - I have various media (movies, music, books, TV shows etc.) related to a single series in one folder. Those should also be in my main music etc. folders. Again tagging with the series name to make a virtual "series" folder would be nice, but symlinks solve that and it happens infrequently enough that it isn't an issue.

Other than those edge cases, I'd say most of my data fits pretty well into a hierarchical structure.

jandrese3• 10 years ago

This is an interesting idea, but it seems like an awful lot of work on my part to go through and organize and tag every file. While some of it could be automated (pulling ID3 tags out of MP3s), a lot of it seems to depend on me figuring out good names for everything I make.

The biggest problem is that I have to figure out which tags are going to be useful to me in the future and where to add them. This is relatively easy for music (but even there can explode in complexity depending on how granular you want to be), but more difficult for things like photos or papers.

IMHO, fully general tagging systems never work because the complexity explodes as the number of potential tags increases. You need to narrow the scope down to a specific domain so your tags can be limited to human scale.

bossman7022• 10 years ago

This is really complicated

bossman7022• 10 years ago

Hello

awesomerobot• 10 years ago

This just seems like a different problem.

qwertyuiop9242• 10 years ago

This is actually kind of akin to BeFS. Although BeFS had greater capabilities in some ways, being an actual filesystem. For the uninitiated, BeFS was the native filesystem of BeOS, and allowed for metadata attributes that allowed for querying and indexing capabilities akin to an relational DB. Or at least, that's what Wikipedia says.

fh973• 10 years ago

This idea is rediscovered every few years in a new project. I tried this with StorageBox many years ago and even did some UX research in this context. Turns out many users don't like querying only as it they feel they can not search their data exhaustively, and might "lose" some of their data this way.

Also for not tech savvy users, folders have the nice interaction pattern of question-response via menu selection. They see something, click, see something, click, without realizing that they are navigating a folder hierarchy.

Already__Taken• 10 years ago

From my minor tech support experience, users don't understand what they've saved a file as. Search is great but if you don't know what the needle is it gets hard.

JoeAltmaier4• 10 years ago

Not convinced. Why wouldn't a 'tag browser' work just like that too? Even better, since what I'm clicking on are meaningful tags, instead of 'directory names'

toolslive• 10 years ago

Spotlight can do the same thing, no ? https://en.wikipedia.org/wiki/Spotlight_%28software%29

pnathan• 10 years ago

Very cool attack on the problem.

I've been working on ideas for managing filesystems and tools for thought for a while off and on, it's starting to coalesce into a set of design ideas as well as prototypes.

What I won't do is set up a filesystem: I don't think that adds value to me; it mostly sounds technically complex and hard to figure out. And, I think that driving the whole business through manual tagging is a lost cause. Manual tagging can be _useful_, but actual attempts to derive semantic knowledge will be _more_ useful. I have many thousands of documents - manual tagging ain't gonna happen.

I need _semantic_ search and _semantic_ cross-referencing; something like a Xanadu or a (much better) wiki/hypertext system.

tmaly• 10 years ago

I could see where a tool like this would be a huge help to legal and compliance departments.

There is a frequent need to produce documents for discovery purposes. However, you generally have to review the documents or send the to outside counsel first for review before they are produced. This can take hundreds of man hours and cost thousands of dollars.

Building something on top of TMSU could be a great solution for this task.

mywittyname2• 10 years ago

The reason hierarchical structures are used in file systems is because they are a pretty intuitive and, most importantly, generic way of classifying and storing information. For the most part, just about any file you have on your computer can be stuffed into some sort of folder structure.

Specialized files like music, movies and images are a solved problem. iTunes and other software do a great job at organizing this information and making it easy to use. There's also software like Quicken for dealing with the other common mountain of data people have.

What might be useful is a piece of software that is capable of extracting and managing metadata automatically. Think of a tool like iTunes that you feed it a collection of files and it uses some form of ML to extract and create a database build logical ontologies for this data. The big problem with this kind of tool is finding a large, complex dataset that an individual has, but that has not been organized by a specialized piece of software. I doubt these exists in numbers significant enough to justify creating a project.

tl;dr: Directory structures are low-effort, generic, and discoverable ways of dealing with files that are not managed by other applications. It will be hard to improve on them without sacrificing one of those three attributes.

JoeAltmaier4• 10 years ago

I don't think its intuitive at all. We don't ask questions like "What parent folder did I put that document it?" We ask "Where is that document I printed yesterday. The one that I got via email."

With liberal use of tags, and the ability to browse them fluidly, we could ask those sort of questions.

mywittyname2• 10 years ago

My Documents [Sort by Date]

At the top.

The way that one person manages their documents probably isn't going to be the same as another, but generally a person is consistent with all their files.

In this case, it's either a document that you have several types of, in which case you would have an existing folder structure, or it's a one-off that you dump in My Documents along with all your other one-offs. Even if you lose it, file systems all keep date-time information, so you can easily search for the all the files last modified yesterday.

The problem with liberal tagging is that it requires a bunch of up-front effort that you're never going to perform. In this case, if it were an email, you'd just use your email browser (specialized software!) to find the document again based on the things you remember about it (date, source, size, etc).

JoeAltmaier4• 10 years ago

Its been a generation since the hierarchical file system was obsolete. Its lame to have to hang every file on the ceremonial file tree like some Christmas ornament. Hardly any app wants data organized like that.

In fact, nearly every large app does something to avoid it. They create their own representations of a log, or a mail folder, or a document, or an image (and on and on) and manage the details themselves. Because 'file systems' are so lame and underpowered.

This tool begins to help. Creating flexible groupings (tags) resembles a relational database. That's a start. I'd like to replace the OS file system with something like that.

Instead of renaming files when you bring another copy onto your persistent storage, you could just add a version tag to them. Leave the names alone! I can tell my build system (or document store, or mail tool) what version I want to deal with e.g. tag='version' value='2.5'. No collisions any more. No requirement by the 'file system' to mash them into some file tree so they can still be found, but don't collide.

In fact this system can do everything the hierarchical file system can do, and more. Just add the tag 'parent directory name' and voila! You have a file tree (if you want).

vgnanand• 10 years ago

For some reason, I knew this would be about tagging files before reading the article.

jprzybyl• 10 years ago

I can't believe this, but I created something like this in an ad-hoc way. I have a large amount of music, and I needed to be able to transfer them to devices (my phone) based on some sort of tag - I need work-out music, I need driving music, etc. Genres are inappropriate for this. The easiest way to transfer this to the device is have a directory full of links that point to the right files / directories.

So, depending on the need, I will either have two directories (files and tags) or a number of file directories and a tag directory. File directories can have whatever they want, and tag directories have either only tag directories (like workout, driving, etc) or soft links.

Tagging a file / directory is easy - just link to it. Untagging is just as easy. The links don't take up much space, especially next to the music. When I'm transferring the files, I either use a script to make a directory with the links replaced with their file counterparts, or I transfer with something like rsync that can do that itself.

I'm amazed how similar this project is to my own solution. It's nice to have a dedicated script for the whole thing, but the solution itself is very simple, and easy to script with.

This isn't born out of frustration - hierarchical filesystems are perfectly adequate for most tasks. But they have not been trees for a long time - we have links, which let us make any graph we want out of those trees.

e12e2• 10 years ago

TMSU does not alter your files in any way: they remain unchanged on disk, or on the network, wherever your put them. TMSU maintains its own database and you simply gain an additional view, which you can mount where you like, based upon the tags you set up.

I'm not so sure this is a great design decision. Now your tags are only in your sqlite file, and you'll have to work extra hard to get a copy of the relevant tags when you backup/copy etc.

I think storing tags in extended attributes[x], and possibly a separate utility that maintains and index (hopefully shouldn't be needed just for the tags, but might help with a) exposing file-level tags (like ID3, exif, file-type (magic number) etc), and b) allow for automatic organization based on full text and other content-based indexing.

It appears, on a * nix system, the only major reason to stay away from extended attributes (apart from the limit on size of tag data) is NFS. But samba should (AFAIK) work fine with extended attributes.

As far as I can gather, Gnome Beagle is dead, and Gnome Tracker[t] has taken its place. But it's not crystal clear if Tracker will index tags placed in files' extended attributes or not. If I understand correctly, Tracker's own tagging utility, will only place/edit tags in the Tracker database/index. But the indexers will certainly honour file-level tags for some files.

I don't really use full Desktop environments, but some kind of system with inotify support, and a Xapian or similar back-end (like Tracker), does seem like a good idea. It would certainly be nice to see such a system implemented in Go, but I think an architecture along the lines of Tracker is probably worth keeping: A database daemon, an indexer and a set of query/view tools (I'm not a fan of the centralized tag database, though).

Another alternative to Tracker would be Recoll:

http://www.lesbonscomptes.com/recoll/index.html

[t] https://github.com/GNOME/tracker

[x] http://www.lesbonscomptes.com/pages/extattrs.html

Btw, for editing/automating ID3 tags, I recommend "Ex Falso", the tag-editor for Quod Libet (which is an audio player): http://quodlibet.readthedocs.io/en/latest/

padator• 10 years ago

An old solution to this problem was the Logic File System https://en.wikipedia.org/wiki/Logic_File_System (disclaimer: I am one of the author).

35bge57dtjku• 10 years ago

Nice logo. D:

mysecrets• 10 years ago

Reminds me of ReiserFS's original goals.

https://reiser4.wiki.kernel.org/index.php/Future_Vision

prewett• 10 years ago

I feel like I must be missing something, but I feel like tags is just a band-aid. If you have enough files, you will end up with too many tags to manage (think of a tag directory filled with tag files, instead of a home directory filled with actual files), so you'll need hierarchical tags. Or kludge it with "tag/subtag" which looks a lot like a directory. GMail added nested tags, which seems to me like an admission that flat tags is not sufficient.

One problem is that regular people don't know how to organize a hierarchy. General -> specific works really well, but that requires the ability to generalize.

The only use I can see for tags is if you want files to be a member of more than one directory. Other than music, everything I have is generally created for a specific purpose, so tags are not particularly helpful.

caseymarquis• 10 years ago

This is just PDM.I built a system that does all of this, more, and integrates into our company's existing products for managing manufacturing data. While my version only watches the directories you tell it to, it also detects file movement and changes via a service and through the hashing of files. I'd be more impressed if someone built this into a file system directly. Combine advanced tag based indexing with zfs and you've got something impressive.

arnoooooo• 10 years ago

I wish it would support tag hierarchies.

AlphaWeaver• 10 years ago

This reminds me of something that would be useful in implementing Desktop Neo https://news.ycombinator.com/item?id=10932378

TMSU: a tool born out of frustration with the hierarchical nature of filesystems

Related Stories