Tagsistant: a reasoning semantic filesystem for Linux and BSD

http://www.tagsistant.net/component/content/article/10-what-is-tagsistant

by gnosis • 15 years ago

64 39 15 years ago

p4bl0• 15 years ago

I had exactly this idea last year but I never took the time to actually implement it. Nice to see someone did!

mnzaki4• 15 years ago

I was excited about this at first, but I have come to think a filesystem that does away with the tree structure would be rather difficult to deal with or get used to. I will probably end up with loads of files under similar tags and it would make files rather difficult to locate due to the sheer number of them. Maybe the idea just needs a bit of refining?

hsmyers• 15 years ago

Based on what was mentioned in the article, the 'refining' you mention might well be a plug in. Tagging anything can be a significant task---tagging many 'anythings' creates problems of scale. Another thing to consider; if this is in place at the beginning of the information cycle, I suspect that it would be easier than if this is used to convert from older approaches. Still for all of that, interesting idea/work.

andrewflnr3• 15 years ago

I think just a bit of refining may be required. The filesystem would probably come with a few tags already, so there's less chance of confusing the basic ones. With an easy-to-use tag manager that keeps track of all the extant tags, and the ability to do batch operations across all tags, I think it would work fine. The tricky part will be integrating with systems used to hierarchical directories.

Instead of having operators like AND be path components, I would have the path separator itself be an and operator, or use & and | to just make logical expressions. As long as you're throwing out the hierarchy, you may as well throw out the hierarchical syntax.

mnzaki4• 15 years ago

Having operators be part of the path itself actually struck me as genius. This way tagfs is 'backwards compatible' with current file managers and you can 'browse' the filesystem quite naturally. Having the operators as path separators would require changes to the filemanagers and the host of tools we already use to work with our filesystems.

masterzora3• 15 years ago

This is actually something I've been wanting for a long, long time. Obviously I'm not planning on using this for / or anything system-related, but for my media archives &c it's perfect. Taking movies for example, I've found most people I know think an alphabetical approach is the way to go. This is useless, however, if I don't know what I want to watch. With the tags I can easily take tag in genres, directors, actors, etc and have my filesystem help me choose a film.

There is on major thing that I've considered would be an awesome addition to a system like this: automatic tagging from the metadata. It'd be awesome if my movies automatically got tagged by, say, resolution and length, so I wouldn't have to bother with such things.

mnzaki4• 15 years ago

Same here, I want something similar to manage my photo collection. The current implementation actually can have plugins that act on specific file types, so automatic tags are just a few lines of code away.

But then what advantage would this system have over a dedicated media manager or the several semantic desktop projects? The only advantage I see is allowing traditional tools (unix utilities, conventional file managers, even 'open file' popups) to take advantage of the system without any extra effort on their side. Though wouldn't dedicated tools, say a full blown semantic desktop, allow for much better integration and usability?

roryokane• 15 years ago

Tagsistant already supports automatic tagging from the metadata such as you have described. The linked article says it supports plugins that can “extract tags from file contents” in the second-to-last paragraph. So if you want to tag your movies the way you said, you just have to write a plugin that uses some library to extract that data, or wait for someone else to release a plugin like that.

_urga• 15 years ago

Well done with the "Europe" tag includes anything with the "London" tag idea.

gnosis3• 15 years ago

This idea is called "hierarchical tagging". Searching for that phrase in google should get you some links to other people who've discussed and implemented it.

Someone• 15 years ago

I think that was a joke. Over in London, 'Europe' is across the channel, on the mainland.

TeMPOraL• 15 years ago

One drawback I can think of is that you loose the "permanent paths" to files in some cases. If you start twiddling with relationships between tags, I can imagine how many paths previously stored in software will get broken.

I can also imagine 'identity problems'. Not counting symlinks and hardlinks, the full file path serves as its URI. How can I be sure if /photos/europe/DSCN0001.JPG and /photos/london/DSCN0001.JPG are the same files? What's the file URI here?

mnzaki4• 15 years ago

Another related ramification of doing away with heirarchy and unique identifiers (tree paths) is not being able to have files with the same filename.

Say you have: /photos/london/DSCN0001.JPG and /photos/berlin/DSCN0001.JPG And the relationships: europe contains london, europe contains berlin

Now what does the 'path' /photos/europe/DSCN0001.JPG resolve to?

tx06• 15 years ago

Tagsistant 0.2 does not allow to store two files with the same name, exactly as you say. But Tagsistant 0.4 will! Well, at the little compromise of having a small unique number prepended to each filename.

Tagsistant 0.4 has a broader vision (tagging of entire directories) but is still under development. If you have suggestions or doubts, I'll be very happy to discuss it.

gnosis3• 15 years ago

"If you start twiddling with relationships between tags, I can imagine how many paths previously stored in software will get broken."

Twiddling with tags is no worse than twiddling with directories.

Few people consider it a fault of the design of ordinary filesystems that if you mess with the underlying filesystem layout, software that relied on that layout might break.

The same is really the case for any kind of dependency on a certain kind of organization of your data.

The fault for breaking software that's tightly coupled to a certain underlying organization or layout lies with the software itself (for not tolerating changes) and with the user for making the changes in the first place.

"I can also imagine 'identity problems'. Not counting symlinks and hardlinks, the full file path serves as its URI. How can I be sure if /photos/europe/DSCN0001.JPG and /photos/london/DSCN0001.JPG are the same files? What's the file URI here?"

But symlinks and hardlinks are the critical bit of filesystem functionality that makes ordinary filesystems subject to the very question. So why would you not consider them?

There are various solutions to this problem on ordinary filesystems: first, your tools (like "ls") could show you that a file or directory is symlinked (though you might have to traverse through the parent directories to find out whether there is a symlink). Second, you could also use stat to check the inode of the files in question to see if they're the same.

It should not be difficult to add similar functionality to a tag-based filesystem.

tx06• 15 years ago

I always thought at Tagsistant as an archiving tool, but you are totally right.

But files are also accessible from the archive/ directory where nothing is supposed to change as a consequence of tagging.

Can be a reasonable compromise?

sixtofour• 15 years ago

This is a great idea. Early use would have to be limited, because the rest of the world assumes directories. It would be great if the world ran on tags though.

Bookmark tags is why I'm still using Firefox instead of Chrome. I recently tried Chrome for a week, I really wanted to like it, but bookmark tags is what brought me back to Firefox. I look at my browser as an information manager as much as a reader, and multiple tags per bookmark is my killer feature.

Gmail's imap folders as tags, multiple tags per message, and their exposure of imap to external clients like Thunderbird, is why I was finally able to convince myself to use gmail instead of my hosting provider's email. I confess that I don't use multiple tags as much as I thought I would. But I could!

gnosis3• 15 years ago

"This is a great idea. Early use would have to be limited, because the rest of the world assumes directories."

Tagsistant still uses directories. It's just that the directory names are automatically also usable as tags.

haliax• 15 years ago

Why couldn't this co-exist with a hirerarchical filesystem, maybe via a special subdirectory as procfs does? (With homonym files getting some distinguishing prefix, possibly based on their hierarchical paths.) It seems somewhat like a more sophisticated version of spotlight in that it can handle logical relationships between tags.

janjan• 15 years ago

I'd like to see something similar:

Lately I have thought about a filestorage like this which consists of two parts: 1) some kind of database in which you can put binary files and attach tags to them 2) a FUSE (?) driver which let's create different 'views' on this database which then can be mounted as part of a normal filesystem.

For example this would be nice for music and pdf collections. You could create different 'views' and then go to one folder to see your music/pds sorted by year and then to another folder to see them sorted by author/artist and then to a third one which is sorted by type_of_music/artist/album.

this way you would get the best of both worlds: 1) a powerful database to store and organize binary data and 2) downward compatability since you can just use the command line / bash to 'export' files to mp3 players and so on.

judofyr• 15 years ago

Woah, this pretty much describes my "perfect" file organization tool which I've been thinking about in the last months. I was thinking of combining it with revision support (every version of your file stored) together with Dropbox-like synchronization.

elcron• 15 years ago

I think every version would be a bit much, especially on smaller drives if it had logarithmic diffs i.e. every change today. Every hour for the past week. Before that every day for the past month. etc. It would probably be more disk efficient for people with smallish SSDs, but everybodies "perfect" is different.

tx06• 15 years ago

Tagsistant is not supposed to replace posix filesystems and can't be used in system directories like /bin or /etc.

Tagsistant is a personal tool to organize files (and directories, starting with version 0.4).

It provides a plugin architecture to allow autotagging.

hmottestad2• 15 years ago

Yay...semantic technologies. I hope they make a way of utilizing existing technologies from the semantic web. Like OWL :) https://secure.wikimedia.org/wikipedia/en/wiki/Web_Ontology_...

currywurst• 15 years ago

If you read his notes, he finds that OWL as it currently exists is too cumbersome for his needs.

hmottestad2• 15 years ago

ahh...yes...http://www.tagsistant.net/documents-about-tagsistant/semanti...

I think OWL would fit the task quite well. Yes, right now it may seem like overkill and will probably we somewhat slow. RDFS is an alternative though. Or just using a subset of OWL, like OWL lite.

colanderman2• 15 years ago

I wrote something like this using FUSE and Postgres a couple years ago. The major problem with this approach is scalability: on a large, multi-user FS, you can get TONS of tags to sift through in your root folder.

My solution, which I never got around to implementing, was to have "hierarchical" tags. e.g. "recipes:cookies:chocolatechip" is a single tag for all chocolate chip cookie recipes, all of which have the implied tags "recipes" and "recipes:cookies" (but not "cookies" or "chocolatechip"). The advantage is that your root directory will not be littered with the "cookies" and "chocolatechip" tags when these collections are fairly irrelevant.

This idea can be extended to solve multi-user conflicts (all tags are prefixed with the username) and to make sensible tags for dates -- it makes little sense to tag a file with "2011", "May", and "22", since no-one cares to find all files from the 22nd of every month; but the hierarchical tag "2011:May:22" is perfect for this situation since these files will also be present under "2011" and "2011:May".

tx06• 15 years ago

Did you use an ontology or some kind of "manifest" to bootstrap hierarchical relations between tags?

I'm thinking about splitting Tagsistant into a client and a server to provide a multiuser environment, and probably some ontological foundations are required to coherently organize tagging coming from different users.

colanderman2• 15 years ago

I never got around to implementing the hierarchy. My idea was to have a few basic entries, say, "users", "dates", "MIME-types", etc. which could be prepopulated.

CurtHagenlocher• 15 years ago

Reminds me of WinFS (http://en.wikipedia.org/wiki/WinFS)

simcop2387• 15 years ago

I'd consider doing something like dropping the AND by default.

  i.e. /photos/london/2011

while still having the OR and NOT and any other operations you need. the idea being that at least due to training from hierarchical filesystems the AND would seem rather implied. As far as considering the duplicate filenames i'd add a prefix to them based on the order they were added to the DB (e.g. first one gets no prefix, second one gets a 2_ or whatever you figure out) that way you won't has as many cases where a file's name changes the moment you add in a new one.

tx06• 15 years ago

Tagsistant 0.4 doesn't use AND any longer, but that brought the need for a termination operator, which is '=' so far, but probably will be changed for shell comfort.

The 0.2 query mpoint/rock/AND/seattle/ becomes mpoint/tags/rock/seattle/=/ in 0.4

And about the prefix, Tagsistant 0.4 indeed uses a NNN_ prefix to filenames to allow for duplicated names to coexist.

Tagsistant: a reasoning semantic filesystem for Linux and BSD

Related Stories