Show HN: Tired of the non-portability of my playlists, I wrote my own format

http://universalplaylist.stavros.io/

by stavros • 9 years ago

144 101 9 years ago

universalplaylist.stavros.io

qntmfred• 9 years ago

My own format = https://xkcd.com/927/

rafaelgarrido• 9 years ago

oh man... 10 minutes late!

bb882• 9 years ago

I don't see why this is being downvoted. It's fairly apropos to the subject. And it was the first thing I thought of too when I read the title.

alnitak• 9 years ago

À format that is independent of a specific platform or player has its own merits, besides the scratch your itch part.

With a good plugin support an ecosystem to allow interoperability of playlists becomes possible.

StavrosK35• 9 years ago

I don't plan on creating a standard that will end all others, though. I just want to move from Google Play to my local library.

pavel_lishin• 9 years ago

The introduction sounds much more ambitious than your comment:

The Universal Playlist Format (UPF) is a data format for music playlists. It aims to be more flexible and complete than existing formats. A Universal Playlist (UPL) will allow better description of the tracks it contains, as well as enable added functionality in music players.

I am really curious about moving from Google Play to a local library, though - can you say more about that? GP has been losing tracks, slowly but steadily, or mislabeling them horribly.

Jaruzel• 9 years ago

I think we all thought of this XKCD.

However, it has been said, that there are no new stories; they've all be written, and that everything worth inventing has already been invented.

If we actually believed that, there'd be no new books, and stuff like graphene, and CRISPR wouldn't exist.

I say, let everyone re-invent the wheel, one day someone might actually get anti-grav ones working - You never know.

a2tech• 9 years ago

I'd love something like this, but I don't see any upside for iTunes or Spotify to support it. Anything that makes it easier for a customer to switch to a competitor is something they're going to resist.

snissn• 9 years ago

Ha that reminds me to cancel my Spotify, thank you!

StavrosK35• 9 years ago

They have plenty of upside to allow import through this format, as you can just upload your Winamp (yes, I'm old) playlist to Spotify and immediately migrate to their service. I'm hoping there'll be enough user demand that they'll choose not to appear evil and support export as well.

supermdguy• 9 years ago

This looks interesting. Do you have any plans to create converters to convert to UPL from other formats?

StavrosK35• 9 years ago

Yes, right now I'm collecting ideas (there have been some very good ones in this thread), and I plan to write a few small converter programs to convert from M3U/PLS, as well as some plugins (I'm thinking about beets, currently).

Any help would be appreciated!

enimodas• 9 years ago

why does the example have md5 and sha1?

StavrosK35• 9 years ago

Good catch, I had them there initially but decided to play it safe and avoid the broken hashes, so I removed them from the list but forgot the example, thank you.

discreditable• 9 years ago

What's wrong with xspf? Couldn't you just extend that with more attributes (which legacy apps could ignore)?

https://en.m.wikipedia.org/wiki/XML_Shareable_Playlist_Forma...

StavrosK35• 9 years ago

Nothing's wrong per se (and it has roughly the same goals as mine), it's just that it's pretty much the XML equivalent of M3U, and doesn't have any provisions for stronger identification (it relies on titles), at least as far as I know.

I would really like the MBID to be the main way of identifying songs throughout the industry, and possibly the AcoustID fingerprint, as that's more specific. I think it would be fantastic if each song had its own UUID that computers could refer to, and this spec tries to incorporate that.

Since I don't really gain anything from reusing its format (most players won't support the strong IDs), I thought I might as well create a new format that was better suited to this task.

lucasgonze5• 9 years ago

XSPF is a format to enable sharing, AKA universality. It does this by defining a list of metadata fields to be used for resolving each track in the local context of the listener.

The UPF "ids" field maps to the XSPF "identifier" field: http://xspf.org/xspf-v1.html#rfc.section.4.1.1.2.14.1.1.1.2

XSPF is not M3U in any way. From the spec: http://xspf.org/xspf-v1.html#rfc.section.3.4

3.4 Content resolver

On a surface level you can use XSPF like any other playlist format. Drop a bunch of filenames into an XSPF document, prepend "file://" to each, and you're ready to go. Under the surface there is much more.

The guiding design principle was to separate the functionality of a catalog of files from the functionality of a list of songs. Most software music players on the PC have some sort of cache for file information. This cache stores a list, or catalog, of available files and metadata from ID3 tags and other sources. XSPF is not a catalog format. XSPF exists only to say which songs to play. Almost everything in XSPF is for the purpose of answering the question which resource, rather than the question what is this resource.

If XSPF is not a catalog format, what is it? XSPF is an intermediate format. We expected a new kind of software called a content resolver to do the job of converting XSPF to a plain old list of files or URIs. A content resolver would be smart enough to keep your playlists from breaking when you move your media from /oggs to /music/ogg. It would be able to figure out that a playlist entry by the artist "Hank Williams" with the title "Your Cheating Heart" could be satisfied by the file /vorbis/hankwilliams/yourcheatingheart.ogg. It might even know how to query the iTunes music store or another online provider to locate and download a missing song.

The content resolver maintains the catalog of your songs in whatever format it prefers. It might use a flatfile, a file in the Berkeley DB format, or a SQL database. It might use only ID3 metadata, but it might also know how to query MusicBrainz or another metadata service.

All XSPF user agents are content resolvers, in that they have complete leeway to turn the contents of a track element into a specific set of bytes.

3.5 Fuzzy names

Any given track can be identified in a number of ways. We provided means for absolute identifiers like URIs, filesystem paths and secure hashes, but also for query-based identifiers — free text fields like artist and work title and numeric fields for song length, all of which together should be enough for a good content resolver to turn into files.

jbk• 9 years ago

Yes, please!

We have enough playlists formats to support already (and most of them are half-baked/half-broken already). There is no good reason to not reuse and extend xspf.

(Last time I counted 16 major playlists formats in VLC...)

StavrosK35• 9 years ago

Isn't one good reason the fact that you avoid the whole confusion of "I imported my XSPF playlist with all my IDs but my player didn't find any songs!" "Oh, that's because your player only supports XSPF 1, not 1.1"?

hawski• 9 years ago

Are relative paths supported?

Probably in this day and age it would have better chance at adoption if it would be a JSON file. It's just network effects.

StavrosK35• 9 years ago

Not yet, but great idea, I've added it, thank you!

YAML is a superset of JSON, so the file can look like JSON if you want, although that's not mandatory. I don't think there are many languages that don't support YAML these days, and one of my goals was to make the format not-entirely-unreadable by humans.

masklinn2• 9 years ago

I don't think there are many languages that don't support YAML these days

YAML is a big and complex format though, libyaml's header is bigger than the entirety of libtoml.

I think hand-editable playlists are a good idea, just not a fan of yaml.

hammerandtongs3• 9 years ago

In practice there aren't many languages that support anything more than yaml 1.0 from my last look a few years ago.

Ie yaml isn't nearly as supported as it looks once you start actually trying to use it between languages with anything more than 1.0 as most of the purported library support seemed to be roughly 2009 abandonware.

Yaml also precludes easy https://www.sqlite.org/json1.html and postgresql jsonb etc. Not to mention trivial js consumption of the file.

Superset or not, Yaml automagically introduces bugs to and from json through the magic powers of more lines of code involved.

Most projects would have a hard time not having a json library these days? I've not done a survey of music apps though.

In practice, as someone that has been vim/emacs for a few decades I'd put a low priority on yaml vs json text editability. They are about the same for changing a few characters and I'd probably use orgmode+ python to do anything non-trivial but not full program(and probably drive the transform via json data from somewhere...).

In 2017 yaml doesn't seem tasteful for this format.

zeveb3• 9 years ago

Probably in this day and age it would have better chance at adoption if it would be a JSON file. It's just network effects.

And here I was feeling badly for thinking that it'd be better as S-expressions! I wasn't going to post that, but since you already did, here's the same example in the original YAML:

    ---
    format: UPL1
    name: Favorites
    id: 2b43009f-d6a6-4f00-8533-09a9a73d8b54
    entries:
    - artist: Anciients
      title: Following the Voice
      duration: 408.764081632
      ids:
        sha2: e577cce68a69735acccd5d8603b3e663f6aa5bc9
        sha3: e577cce68a69735acccd5d8603b3e663f6aa5bc9
        mbtrackid: b00a2b97-53f1-485a-9121-1fe76b55e651
        filepath: Anciients/Following the Voice.mp3
        uri: nfs://example.com/music/ftv.mp3

as JSON:

    {
        "format": "UPL1",
        "name": "Favorites",
        "id": "2b43009f-d6a6-4f00-8533-09a9a73d8b54",
        "entries": [
            {
                "artist": "Anciients",
                "title": "Following the Voice",
                "duration": 408.764081632,
                "ids": {
                    "sha2": "e577cce68a69735acccd5d8603b3e663f6aa5bc9",
                    "sha3": "e577cce68a69735acccd5d8603b3e663f6aa5bc9",
                    "mbtrackid": "b00a2b97-53f1-485a-9121-1fe76b55e651",
                    "filepath": "Anciients/Following the Voice.mp3",
                    "uri": "nfs://example.com/music/ftv.mp3"
                }
            }
        ]
    }

And as S-expressions:

    (playlist
     (format UPL1)
     (name Favorites)
     (id 2b43009f-d6a6-4f00-8533-09a9a73d8b54)
     (entries
      (entry
       (artist Anciients)
       (title "Following the Voice")
       (duration 408.764081632)
       (ids
        (sha2 e577cce68a69735acccd5d8603b3e663f6aa5bc9)
        (sha3 e577cce68a69735acccd5d8603b3e663f6aa5bc9)
        (mbtrackid b00a2b97-53f1-485a-9121-1fe76b55e651)
        (filepath "Anciients/Following the Voice.mp3")
        (uri nfs://example.com/music/ftv.mp3)))))

I wonder which in general people think are prettier.

StavrosK35• 9 years ago

For me, it goes YAML, S-expressions and JSON, in order of prettiness.

chadlavi• 9 years ago

not very constructive suggestion: an h2 with a recursion joke, something like "YAML As Music List"

ianwalter• 9 years ago

This is such a good idea. If nothing else, third party tools that people make to transfer playlists between platforms could standardize on this and, in doing so, would reduce duplicated efforts. After trying (and being dissatisfied with) EVERY music service, I'm currently trying to consolidate back to Apple Music. Manually. It's awful.

StavrosK35• 9 years ago

Since this is coming along rather okay, I think I will write a converter this weekend. I aim to have it accept an M3U/PLS playlist and the corresponding files and write out a UPL.

The opposite step would be a bit harder, I'd have to index all music files in a library and convert the UPL to PLS, which might be too much work for a simple tool, but I'll see if it can be done quickly and dirtily. That would be a nice idea, `upl2pls -i favorites.upl -o favorites.pls` and your favorite player is ready to play the files.

rocky11382• 9 years ago

I've never had a problem with my Winamp playlists going from one PC to another. I just double click them and the music plays.

Moru• 9 years ago

And if you mive the library you do a search-replace on the path :)

rocky11382• 9 years ago

I never move the library. All my stuff is stored in syncthing across all my devices.

partycoder• 9 years ago

XKCD predicted this. https://xkcd.com/927/

cmpb• 9 years ago

I'm a little confused why '3.3.1. entries' is a sub-bullet of '3.3. id'. Perhaps it's just a typo?

StavrosK35• 9 years ago

Yes, the entry names are automatically done by a hacky module I wrote, I'll fix it, thank you.

woodruffw• 9 years ago

This is great, thanks for creating and sharing it. I've been working on my own music player of sorts [1] that uses YAML to store playlists, and I'm going to look into switching to this.

[1] https://github.com/muzak-project/muzak

StavrosK35• 9 years ago

Oh, fantastic! Please let me know if you need help with anything or have any feedback. An issue in the project repo is great, or just email me directly (email in profile). Thanks!

_pmf_• 9 years ago

Directories of numbered symlinks, my friends.

autocracy2• 9 years ago

Removal performance of large directories is very bad. Reading large directories can result in lots of seeks. Directories are not suited to this at scale, and symlink directories definitely aren't portable.

autocracy2• 9 years ago

A side note: at least in python, I benchmarked MsgPack as 3 times faster than JSON, and a whopping 850 times faster to read than YAML. It seems unlikely that people will ever be editing this file by hand, so for unstructured data, especially where playlists can get very large, I suggest MsgPack over YAML.

BUT... you have a defined schema, so it's probably to your advantage to use a storage format with a defined schema: ProtoBuf or Thrift. That would mean somebody trying to use your code would already have generated objects in their language.

As for the hashing algorithm, this is a good use for MD5 -- cheap and fast. You're unlikely to be concerned about somebody actively trying to generate the same checksum for two music files. For non-security (integrity verification) purposes, MD5 is still very appropriate.

StavrosK35• 9 years ago

Thank you, I agree. If I'm going to use JSON, I might as well use ProtoBuf.

About MD5, I was worried about a case where a service that serves user-submitted files would be exploited by MD5 collisions, leading users to open files that might exploit decoder bugs to execute code. Far-fetched, I know, but the tradeoff didn't seem worth it. I'm not married to that decision, though.

nbadg• 9 years ago

The question of hash usage made me think of an alternative approach -- what about some kind of audio perceptual hash? P-hash has support for audio hashes [1] (at least it claims to, but I've never used it). The metadata is useful, sure, but coupling it with the playlist seems like a bit of a strange design choice, if it could be avoided. In my mind, an ideal world would have two databases (or equivalent): one for metadata -> perceptual hash, and one for playlist -> List[perceptual hashes].

The downside of course is this requires pre-calculation of the p-hash for every track to use. But I can't think of a music application that doesn't require some kind of "library loading" step, so perhaps this could be accomplished then?

Of course none of this mitigates your concern with decoder bugs resulting in RCE, but I think that's probably best handled elsewhere (for example, sandboxed upload validation in your hypothetical user-uploaded-files service).

[1] http://www.phash.org/

rout39574• 9 years ago

You can apply a schema to JSON, too.

Can you hear us underneath that pile of bike sheds?

jstimpfle• 9 years ago

If I understand correctly, you want to make playlists portable by associating each item with enough metadata to allow identification even when some metadata, such as filepath, changes.

Playlist and metadata are separate things, and coupling them is not good from an architectural perspective (redundancy/denormalization). The two can still be related, even in a single file: relational database. (plug: I'm working on http://jstimpfle.de/projects/wsl/main.html)

Sketch to make more clear what I mean:

    {
      "metadata": {
        "buss_und_reu": {
          "filepath": "Bach/Matthäuspassion/Buss_und_Reu.mp3"
          "composer": "Johann Sebastian Bach"
        },
        "highway_to_hell": {
          "filepath": "ACDC/Highway_to_Hell.mp3",
          "artist": "ACDC",
          "sha512": ...
        }
      },
      "playlists": {
        "mypl": [
          "buss_und_reu",
          "highway_to_hell"
        ],
        "mypl2": [
          "buss_und_reu"
        ]
      }
    }

This is more normalized, less redundant. Of course other schemes are thinkable, for example storing each playlist in its own file, but this way metadata and playlists will more quickly get out of sync.

The library I'm working on allows automated conversions of such hierarchical data to/from relational databases, so data can be viewed, and even stored, as JSON, while integrity is checked based on an relational schema - something like

    % TABLE Track trackid
    % TABLE Filepath trackid filepath
    % TABLE SHA512 trackid sha512
    % TABLE Title trackid title
    % TABLE Composer trackid composer
    % [Declare also domains, keys, and foreign keys here...]

StavrosK35• 9 years ago

Hmm, yes, this is clearly a better approach, architecturally. However, I don't know whether it's worth the tradeoffs, given that it will make the file harder to parse, and playlists tend to have each song only once, so the added complexity seems to come at no benefit...

It's definitely a better approach for a library, though, and it's how one would model the data in a database as well.

TTPrograms• 9 years ago

I think a few people have open-sourced various audio fingerprinting algorithms. Perhaps we should standardize around one and enable real UUID for audio files? That seems to be the missing piece for a lot of the novel functionality you're looking to hit. Otherwise you're stuck doing fuzzy pattern matching on metadata, which seems tricky especially considering possible variation in live recording etc.

StavrosK35• 9 years ago

I don't think the two are different goals. I would very much like standardization around the MBID and AcoustID fingerprint, and my playlist format is just a playlist that uses that. There's no way to use those with a PLS file, for example, even if they were standard and ubiquitous.

zeveb3• 9 years ago

I think rather than having a single URI attribute and other identifiers, you ought to allow multiple URIs, and make all other identifiers just be URIs (since, after all, a URI is a Uniform Resource Identifier, and music files are resources).

You can represent hashes using the named-information URI scheme (https://tools.ietf.org/html/rfc6920), e.g. ni:///sha-256;u88lYWn4xAlto-6Bs79KHHYDAu28US71ui5Be6C-ZVw.

Your filepath could be file:///Anciients/Following%20the%20Voice.mp3; your mbtrackid could be musicbrainz:b00a2b97-53f1-485a-9121-1fe76b55e651 (since I don't think an authority makes sense in this case).

This would also permit multiple entries for certain types of identifier, which doesn't make so much sense for hashes (although it's possible for the same recording to have two different MP3 encodings, so … maybe it's not crazy), but would be nice when e.g. a song is found in multiple places in the filesystem, or can be retrieved from multiple locations.

lucasgonze5• 9 years ago

That is exactly the approach taken in XSPF: http://www.xspf.org/xspf-v1.html#rfc.section.4.1.1.2.14.1.1....

Notice that the cardinality is "zero or more."

StavrosK35• 9 years ago

This isn't too different from the current format, except the URI method is pushing the type into the URI, rather than into the name of the key.

I do wonder about what you said, though: Would it be useful to be able to specify two different file paths or URIs for the same item? I'm leaning towards no, because, when clicking on an entry, you want it to start playing, so the choice is made when adding the file, rather than when playing it. The problem with multiple file paths is that both have the same weight, so how do you choose?

I guess you'd choose the same way you choose between the formats already, based on whatever is available/found.

zeveb3• 9 years ago

This isn't too different from the current format, except the URI method is pushing the type into the URI, rather than into the name of the key.

Yup, which I think is good, since URI schemes already exist for many identifiers, and more can be added.

Would it be useful to be able to specify two different file paths or URIs for the same item?

I think so. Maybe one is one removable medium and the other is on another.

The problem with multiple file paths is that both have the same weight, so how do you choose?

If they have equal weight, does it matter which you choose?

StavrosK35• 9 years ago

You have swayed me, I have updated the spec, thank you!

djrogers• 9 years ago

Is the lack of portability really due to the unavailability of a (or yet another) standard format? Seems it's far more related to the music player decisions about what it's playlist format should be...

StavrosK35• 9 years ago

How so? How will my music player know that "Dark Tranquillity - The Gallery" from Google Music is actually thegallery.mp3 on my local drive?

geocar3• 9 years ago

Do you know about magnet uri[1]?

magnet:?dn=Anciients%2FFollowing%20the%20Voice&xt.1=urn:sha1: e577cce68a69735acccd5d8603b3e663f6aa5bc9&xs=nfs://example.com/music/ftv.mp3&x.dur=408.764081632

[1] https://en.wikipedia.org/wiki/Magnet_URI_scheme

StavrosK35• 9 years ago

I do, but it doesn't seem directly applicable to this case. It's similar, but it's better suited to torrents and things like that.

geocar3• 9 years ago

Providing enough metadata to recognize the file different ways, is exactly what you're doing.

Magnet already does this.

it's better suited to torrents and things like that.

I don't understand what you mean by this.

rsync• 9 years ago

You know what always, always works ?

Copying the actual songs into a directory that becomes "the playlist".

It's very inefficient and inelegant but after the 100th car stereo I encountered that couldn't make sense of the most basic m3u file, I just gave up ... my "playlists" are just more copies of the actual songs.

Problem solved.

choward• 9 years ago

You could combine both solutions and process a playlist file to put copies in a directory whenever you need to actually use the playlist.

ibizaman2• 9 years ago

That's where fuse really shines. For example (note I never used it) https://github.com/rtokarev/m3ufs2

StavrosK35• 9 years ago

I did that too, and then uploaded those files on Google Music, and now I want to get them back (with the songs I've added since), and... oops.

ibizaman2• 9 years ago

You can avoid wasting disk space by using symlinks/hardlinks. And I must agree, a filesystem interface is always neat.

lutusp• 9 years ago

Title: "Tired of the non-portability of my playlists, I wrote my own format."

Subtitle: "The Universal Playlist Format"

Analysis: https://xkcd.com/927/

skookumchuck• 9 years ago

This is not in the least bit surprising. I download financial information from my accounts in the form of .csv files, but have never found any accounting software that will import those files reliably. I finally had to write my own csv parser and then tweak it for each wretched csv data provider to get things to work right.

For example, download credit card transactions as csv. Import into Quicken. Quicken decides that skookumchuck is the payee on all the transactions. Arggh. This is 2017 fer cryin out loud.

StavrosK35• 9 years ago

I'm currently writing a small converter utility to convert PLS/M3U/etc playlists to UPL and back, and to serve as a reference implementation/proof of concept. If you want to follow development, feel free to star this repo:

https://gitlab.com/universal-playlist/pls2upl

You can also star the repo of the spec website itself:

https://gitlab.com/universal-playlist/universal-playlist-spe...

adamdecaf2• 9 years ago

I was starting to write one too! I've got a couple of commits I can share if you're interested.

StavrosK35• 9 years ago

Oh, definitely, what's it like?

joshka• 9 years ago

One of the key things I've looked for in a playlist format for many years is the ability to handle files that contain multiple tracks. My main example of this is continuous DJ mixes that are a single file. You might also see this in live recordings of performances where the interaction with the crowd between each song.

These single file media items generally have 1 or more Album Artists (e.g. the DJ) with each track in the performance blending into the next. Each track may have an associated Artist and Track name. Conventionally the best approach to solving these is to use a Cue sheet and a plugin for whatever media player that can load these simultaneously. Alternatively (though rarely) a container format that supports chapters / sub streams (e.g. mka) can be used.

I don't see a secondary (CUE sheet) format as being the right solution to this problem. To me a playlist and a list of tracks played in a performance are kind of the same thing and should be handled by the playlist.

My proposed extension to the UPL format is to allow each playlist entry to specify a part of the identified file. Now there's two possible ways of doing this:

1. Sub items under the entry (e.g. chapters)

2. Each entry specifies a start and end time (or duration)

You mention at [1] that this format is about display, I feel this fits within that idea by being something that is otherwise difficult to display using any other available info (ID3 tags etc.)

[1] https://gitlab.com/universal-playlist/universal-playlist-spe...

jason_s• 9 years ago

+1, I was going to suggest the same thing. Or at least the ability to note multiple files which are supposed to be chained together in order. It pisses me off when Shuffle doesn't handle these properly:

- Pink Floyd, Brain Damage followed by Eclipse

- Queen, We Will Rock You followed by We Are the Champions

- Yes, Long Distance Runaround followed by The Fish (Schindleria Praematurus)

- the different movements of Beethoven's Fifth Symphony

StavrosK35• 9 years ago

Very good idea, thanks! I went with #2, as I feel that it's more generic and easier for players to implement.

MobesMobes• 9 years ago

It's not trying to solve the exact same problem, but m-Tags is also using metatags stored in a seperate file. http://www.m-tags.org/

Show HN: Tired of the non-portability of my playlists, I wrote my own format

Related Stories