Thor – A Project to Hammer Out a Royalty Free Video Codec

http://blogs.cisco.com/collaboration/world-meet-thor-a-project-to-hammer-out-a-royalty-free-video-codec

by TD-Linux • 11 years ago

422 164 11 years ago

blogs.cisco.com

jngreenlee• 11 years ago

"We also hired patent lawyers and consultants familiar with this technology area. We created a new codec development process which would allow us to work through the long list of patents in this space, and continually evolve our codec to work around or avoid those patents. Our efforts are far from complete, but we felt it was time to open this up to the world."

This burden is becoming far too great, when this is the cost necessary to achieve innovation.

coldpie• 11 years ago

Well put. It's crazy that they're pumping that much money paying lawyers in order to avoid using work that someone else has already done. They're forced to waste money and engineering talent in order to re-invent a technology due to our fucked up IP legal landscape.

tokenizerrr• 11 years ago

Not just re-invent, since if you happen to invent something by yourself the same way as it happens to have been patented then you're still in trouble.

serge2k• 11 years ago

Or they could pay the people who invented it already.

oh that's right. The whole point is they don't want to.

smithkl42• 11 years ago

Amen. As Thomas Jefferson (correctly) described patents, they're a part of positive law, not natural law. In other words, their only justification is pragmatic, not moral. You can't "own" an idea the way that you can own a couch or a car. We allow for (temporary) patent protection because it's supposed to encourage innovation and help our economy. If it doesn't - and it's clearly reached the point where it hinders rather than helps innovation - then we need to change things.

manachar• 11 years ago

To be clear, because some forget this part of patent law, the encouragement towards innovation was to encourage people to share and build on a communal set of ideas. The temporary monopoly on ideas was the carrot to get people to register their ideas in a central location (patent office) rather than lock ideas behind closed doors and secrecy.

Too many people think the carrot of the temporary monopoly was the point of the patent office.

In other words, we know that the community of ideas gets better with more sharing and building off of other people's ideas. As a society we decided to make laws to encourage this sharing. As a result, technological progress was immense.

Sadly, the current situation seems to strongly suggest that we may need to find better ways to encourage sharing (Open Source has been great for much of software).

happyscrappy• 11 years ago

Tesla(edit:SpaceX) is not bothering to file patents because it is just a blueprint for countries like China who quickly build knockoffs on someone else's R+D dime.

adultSwim• 11 years ago

Are we saying you can no longer commercially develop a video codec?

If it's so good for the industry, the patent holders could just choose to make it free. Everyone else could choose to :gasp: pay.

I actually agree with your point. I just want to make it clear what that means. When it comes to patents, to me a compression algorithm is the closest thing in software to a mechanic device. At that level could patent equivalent ideas as hardware. Are we saying no more patents on mechanical inventions or hardware?

For a long time I've thought that the patent office letting crummy patents through as being a separate issue than having patents at all. It also never made sense to separate out software patents from anything else.

rhino369• 11 years ago

Natural law doesn't exist. It's essentially a religious belief Jefferson had.

harryh• 11 years ago

Hate to break it to you, but your treasured natural laws are just as man made is the positive law that you dislike.

We allow for (temporary) ownership of property because it's supposed to encourage benefits to society of various varieties. If it doesn't - and it's starting to become clear that, say, inequality in the US is becoming too high - then we need to change things.

throwawaykf053• 11 years ago

As has been said elsewhere, this is not the cost need to achieve innovation, this is the cost needed to avoid paying others who have already invented something before.

And as often it is with patents, the outcome is more invention. All those workarounds are also innovation.

ken474• 11 years ago

Many readers would take issue of your use of the word "invented" in this context. Was the Pythagorean theorem also "invented" in your opinion?

donpdonp2• 11 years ago

Didn't we already go through this with VP8/VP9/WebM?

JustSomeNobody2• 11 years ago

I was thinking the same thing.

TD-Linux13• 11 years ago

VP9 is a good choice if you want a royalty free video codec right now. NETVC is shooting for the next generation. In addition, the goal is to get the video codec standardized at the IETF.

NETVC participation is open to anyone though, so it is possible more players will show up.

jerf2• 11 years ago

"NETVC is shooting for the next generation."

What makes a codec "next generation"? I assume, broadly, it involves trading off more yet more computation for a tighter encode? (As nearly an embarassingly-parallel problem, video coding continues to get faster with more silicon even if serial performance is stagnant.) What kind of gains can we expect from the "next generation"?

All honest questions, BTW. Links welcome, though something focused on this question and not just a laundry list of features with no reference to the past would be preferred.

donpdonp2• 11 years ago

Okay they maybe the link text should say "Thor – A Project to Hammer Out the Next Royalty Free Video Codec".

I'd be happy to see VP9 actually get deployed on any significant scale before putting another new codec into the mix.

dharma12• 11 years ago

Seems to me the success will depend on the quality and whether chip manufacturers will embrace this for hardware encoding/decoding. Right now looks to me like h265 is the winning horse

masklinn3• 11 years ago

Thor is part of the NetVC effort, tentatively a royalty-free successor to HEVC.

indolering4• 11 years ago

Are they? I thought they would be scared off by the existence of two distinct patent royalty groups.

yjm• 11 years ago

i wonder how many orders of magnitude slower this one will be compared to x264. vp8/9 was like 9x slower last time i checked

TD-Linux13• 11 years ago

It is currently quite a bit slower, but the goal is to make a codec fast enough for real time communication use.

VP9 is still about 9x slower than x264, but yields the same quality at half the bitrate. You can set VP9 to run a lot faster, but you'll lose some of the bitrate advantages. Still, VP9 is practical for a lot of applications, such as Youtube.

bitmapbrother• 11 years ago

VP9 being 9x slower than x264 is hard to believe. Do you have a citation?

mappu3• 11 years ago

VP9 producing the same quality as x264 at half the bitrate is hard to believe. Do you have a citation? Which `--preset` for x264 are you basing this on?

bobajeff2• 11 years ago

So... this is a separate project from Daala which Cisco also works on. Is there a story here?

CUViper• 11 years ago

*Daala - but yes, I was wondering the same thing...

bobajeff2• 11 years ago

Oops, I somhowe missed it good catch.

qrmn• 11 years ago

Both Xiph's Daala and Cisco's Thor projects are contributing to the NetVC Working Group at IETF - https://datatracker.ietf.org/wg/netvc/charter/ - to attempt to create a new, competitive, royalty-free video codec - in much the same way as Skype contributed SILK and Xiph contributed CELT to the audio codec WG to create the truly excellent Opus audio codec.

We'll see what comes out of it!

s9w• 11 years ago

This seems like fantastic news after the HEVC patent disaster.

Has anyone tested this or has more information on the performance/quality vs other codecs?

TD-Linux13• 11 years ago

I have, at my website (for objective metrics) http://arewecompressedyet.com/

Summary is that Thor is performing at a slightly better level than Daala and worse than VP9 or H265. But it's also missing a lot of features right now, and the encoder is only tuned for PSNR.

JustSomeNobody2• 11 years ago

I am sure that some entity holds a broad enough patent that all your bases will belong to a Texas court.

saidajigumi2• 11 years ago

And that's the real problem. Heck, it doesn't even need to be an NPE, it just needs to be one of the patent holders they're "avoiding" who wants to fire up some litigation.

They don't even need to be able to win. An existing "legit" patent holder might choose to simply throw lawyers at as a tactic to delay or defeat a potential competitor. In that case, it comes down to a cost/benefit analysis for the would-be litigator.

TD-Linux13• 11 years ago

Certainly the risk is better with a royalty-free video codec, though. In the case of Daala, the goal is to be sufficiently different to avoid these broad patents, too.

Also, any companies contributing to the NETVC standard are required to declare IPR, which is not the case for MPEG standards.

TD-Linux13• 11 years ago

This risk is there for pretty much any engineering you might want to do. The point here is to make a royalty-free codec, with as much IPR defense as possible.

halosghost2• 11 years ago

Actually, I'm still rooting for Daala (from Xiph.org, the same folks that did so well with Opus). It's still a long ways away from being finished, but their work is awesome and I've been following it for a while now!

Either way, having another effort competing to make a great format is not a problem. Here's hoping it goes well!

dharma12• 11 years ago

looks interesting, Opus is great

derf_• 11 years ago

Hello, I'm the Daala tech lead.

One of the things that made Opus a success was the contributions of others. We certainly don't have a monopoly on good ideas. We'll take pieces of Daala and stick them in Thor and pieces of Thor and stick them in Daala, and figure out what works best. Some of that experimentation has already begun:

https://github.com/cisco/thor/pull/8

https://review.xiph.org/874/

https://www.ietf.org/proceedings/93/slides/slides-93-netvc-5...

Because none of us have a financial incentive to get our patents into the standard, we're happy to pick whatever technology works best, as long as we end up with a great codec at the end. Hopefully NETVC can replicate the success of Opus this way.

halosghost2• 11 years ago

It's this kind of awesome attitude that makes me so happy whenever I look into a Xiph project. I certainly did not mean to imply that the efforts of the org in OP should be moved to Daala and that competing projects are a bad thing; far from it! Rather, I was just voicing my support for a similar project that has had some great progress so far.

You all have done a great job in the past and I'm always on the edge of my seat to see what incredible things you will do. Keep up the rockin' work!

billconan• 11 years ago

I hope you could have a comprehensive write-up about how codec works.

your article, such as this http://people.xiph.org/~xiphmont/demo/daala/demo1.shtml is the best codec introduction I could find online, but still I couldn't follow, say, the deblocking part.

there doesn't seem to be a book covering this area.

xiphmont• 11 years ago

Yeah, we're going to have to start arguing on what to name the hybrid soon, since 'Vopus' is already taken.

shmerl3• 11 years ago

Is Thor based on similar principles to Daala (i.e. like lapped transforms) to make it useful for merging?

And unrelated question, what will be the name of the merged codec? I hope it won't remain as NetVC, as that name is awful.

masklinn3• 11 years ago

Apparently, as part of the NetVC[0] effort Daala has started experimenting with Thor tech and innovation, Thor's been added to AreWeCompressedYet[1] and they looked into integrating Thor's motion compensation and its constrained low-pass filter into Daala during the IETF93 NetVC hackaton[2] a few weeks ago.

[0] https://tools.ietf.org/wg/netvc/

[1] https://arewecompressedyet.com

[2] https://www.ietf.org/proceedings/93/slides/slides-93-netvc-5... (PDF)

cfluffy• 11 years ago

I'm betting on Daala and Thor. Both the teams involved are motivated by the same goal to make a royalty free codec. I think the odds are good that the IETF standards process will result in combining the best ideas of Daala and Thor and reviewing them from and patent point of view. If the guys doing VP10 wanted to work with us in an open way, that would be great too.

keeperofdakeys• 11 years ago

There was a great talk about Daala at Linux.conf.au 2015 (https://www.youtube.com/watch?v=Dmho4gcRvQ4). It has both general, and technical information about Daala, so it's well worth a watch.

lambda• 11 years ago

Opus was actually a similar combination of two codecs, CELT from Xiph.org and SILK from Skype.

So similarly, there are now two projects, Daala from Xiph and Thor from Cisco, which are both preliminary but being worked on in the same group, so hopefully the best parts can be taken from each to produce a new, better open codec.

fndrplayer133• 11 years ago

Why not throw the weight behind VP9? edit: I actually am curious, this isn't a question pointed at the validity of Thor. I just really want to see a great, open-source standard emerge and see people get behind it.

masklinn3• 11 years ago

Apparently because they're innovating with Thor, according to[0] Daala people are experimenting with Thor tech (IETF 93 was a few weeks ago) and Daala tech is being experimented for Thor[1]

[0] https://www.ietf.org/proceedings/93/slides/slides-93-netvc-5...

[1] https://github.com/cisco/thor/pull/8

mlinksva• 11 years ago

Is Google also contributing to the NetVC effort?

thefreeman2• 11 years ago

I'm no expert, but the article lists VP9 as "proprietary". Which I take to mean not open source, and potentially not free. Though the proprietariness could be a response to the issues they had with VP8 and suspected reactionary patenting.

oconnor663• 11 years ago

VP9 is open source and royalty-free though. I'm confused.

Navarr• 11 years ago

Yay another one!

josu• 11 years ago

So https://xkcd.com/927/?

yellowapple• 11 years ago

The comparison would be more accurate if most of those 14 competing standards are stated to be proprietary and burdened with patents galore; the stated goal of Thor isn't to try to unify a bunch of standards, but instead to actually create a proper non-proprietary standard.

codebeaker• 11 years ago

See also the massively popular, and critical-to-rails Gem https://github.com/erikhuda/thor - naming is hard.

dvt• 11 years ago

Isn't Thor a Norse god, too?

dalke• 11 years ago

And a washing machine, and a family of satellites, and a ramjet engine. https://en.wikipedia.org/wiki/Thor_%28disambiguation%29 . Not listed there, also a special purpose DBMS for small molecule chemistry data.

WaxProlix• 11 years ago

And they're both named after the superhero!

nvader• 11 years ago

Naming may or may not be hard. But the fact that two projects in hugely different contexts happened to pick the same name does not bear that out, nor does anyone confuse that ruby gem with an even more massively popular Avengers member.

Perhaps if you stopped worrying about name collisions with any existing project ever, you might find naming things a little easier.

ChuckMcM3• 11 years ago

<sarcasm mode> No wonder Big Media hates tech, they are trying to take all their money away. </sarcasm mode>

I think this is a great effort, and if you'll recall Google went and attempted to do the same thing with VP8, but found that people could file patents faster than they could release code[1]. I would certainly support a 'restraint of trade' argument, and a novelty argument which implies (although I know its impossible to currently litigate this way) that if someone else (skilled in the art) could come up with the same answer (invention) given the requirements, then the idea isn't really novel, it is simply "how someone skilled in the art would do it." I've watched as the courts stayed away from that theory, probably because it could easily be abused.

[1] Conspiracy theory or not, the MPEG-LA guys kept popping up additional patent threats once the VP8 code was released.

jsprogrammer4• 11 years ago

Why attack novelty instead of non-obviousness? An expert can attest to obviousness, but not necessarily novelty (you shouldn't need an expert for that: simply produce the prior version).

ChuckMcM3• 11 years ago

In my experience (and that experience is limited, I've only participated as an expert witness in two cases that have gone to trial) looking backward through time on "obviousness" is really hard. Once you know how a magician does his trick, its obvious to you, but before you knew it wasn't. Compare that to multiple people from different places trying to solve the same problem came to implement that solution in exactly (or an infringing) way, speaks to the notion that of non-novelty more strongly. And its something that has already happened prior to disclosure, so you cannot argue that the other people were "taught" by the patent disclosure. (one of the tenets of the patent is that it should teach others skilled in the art how to do what it is you're patenting)

Anyway, I'm not a lawyer, and none of this is legal advice or patent advice. Just my thoughts (or perhaps frustrations) on how hard it is to generate deliberately patent free technology. That difficulty suggests to me a way in which patent law could be improved.

jsprogrammer4• 11 years ago

Is it common to characterize BSD licensed software as proprietary? As in, 'Google’s proprietary VP9 codec'?

TD-Linux13• 11 years ago

This refers to the fact that the bitstream format of VP9 has not been standardized anywhere, whereas the NETVC group intends to produce a standard.

JoshTriplett• 11 years ago

Google’s proprietary VP9 codec

That's an odd choice of phrase; it's unfortunate that a press release chooses to disparage alternatives without explanation.

TD-Linux13• 11 years ago

It's referring to the lack of standardization of the VP9 codec, and the terminology is pretty common in standards bodies - it does sound weird and misleading when intermingled with free software though.

anon13852• 11 years ago

Why is that odd? It is pretty much a textbook example of a proprietary standard.

Here are some definitions of "proprietary" as used by members of the FOSS community when talking about standards:

https://news.ycombinator.com/item?id=4634957 (BrendanEich)

"Proprietary" as in "sole proprietor" is appropriate for a project with zero governance, launched by Google after some incubation closed-source, dominated by Googlers.

https://news.ycombinator.com/item?id=9395992 (pcwalton, Mozilla employee and Rust core developer)

In a competitive multi-vendor ecosystem like the Web, public-facing protocols that are introduced and controlled by a single vendor are proprietary, regardless of whether you can look at the source code. NaCl and Pepper, for example, are proprietary, even though they have open-source implementations.

FooBarWidget• 11 years ago

That is a really odd thing to claim, given that there are so many proponents of the MIT license. People claim that MIT is "more free than GPL" because MIT has less restrictions. GPL has more restrictions, and although those restrictions are there to ensure that the same rights are passed on other people, MIT proponents don't buy that argument: they argue that even if someone forks an open source project, slap a GUI on it and sell it as a proprietary product, no freedom is lost because the original is still available. It does not matter that you cannot contribute to the fork. The ability to make proprietary forks is seen as good.

Yet when applied to Google's products, this is suddenly viewed in a different manner? Even if the maintainer does not accept patches, you can still fork it, so no freedom is lost. And it's ok for other people to make a proprietary fork, but not ok for the author to make a proprietary fork? That sounds like hypocrisy to me.

Animats3• 11 years ago

The MP4 patent situation needs another close look. MP4, which was first standardized in 1998, ought to come out of patent soon, if it hasn't already. There are a few remaining patents in the MPEG-LA package, but they're mostly for stuff you don't need on the Internet, such as interlaced video, font loading, error tolerance for broadcast, and VRML. This hasn't been looked at hard since 2011[1] and it's time for a new look. Some of the key patents related to motion compensation expired last April.[2]

It looks like the last patent on MP3 audio decoding expires next month.

[1] http://www.osnews.com/story/24954/US_Patent_Expiration_for_M... [2] http://scratchpad.wikia.com/wiki/MPEG_patent_lists#MPEG-1_Au...

TD-Linux13• 11 years ago

If by MP4 you are referring to H.264, there are still many years remaining on most of the patents. MPEG-LA publishes patent lists, if you're interested to look.

You are right in that there are many other encumbered technologies that have patents expiring soon. MPEG-1 and MPEG-2 video, MP3 and AC3 audio, and several container formats are included. Notably, this is almost all of the technologies required to make a DVD.

vetinari• 11 years ago

MP4 usually means MPEG-4 Part 2 or MPEG-4 Visual (ISO 14496-2). That's the codec that DivX and XviD implemented.

Animats3• 11 years ago

Yes, MPEG-LA publishes lists, and they need to be looked at closely. Most of the patents have expired. When you go down the list, you see things such as US #6,181,712, which has to do with multiplexing two unrelated video streams into one. Broadcasters and cable systems do this, but Internet video does not. US #6,160,849 only applies to compression of interlaced video, which nobody uses on line. US #7,627,041 is about dealing with missing header data due to transmission noise. US #5,878,080 is about backwards-compatible multichannel (>2 channels) audio, also seldom used on-line. US #6,185,539 is for video with overcompressed extra-low-quality audio. So is US #6,009,399.

MPEG-LA has been padding their patent portfolio by dumping in all these patents on little-used features. Until this year, they still had some good patents, such as the ones on motion estimation, which is a hard problem and is needed to make compression work. But those have now expired. What's left looks like it can be avoided as unnecessary for Internet use.

russtrotter• 11 years ago

Wasn't Ogg Theora created under just the same principles? I'm not smart enough in all things codec to know how it stacks up technically, but best I can tell, it's unencumbered.

https://en.wikipedia.org/wiki/Theora

theandrewbailey• 11 years ago

Theora simply can't do what H264 can.

example: http://www.streaminglearningcenter.com/articles/ogg-vs-h264-...

nickpsecurity3• 11 years ago

Deleted a prior comment on "why not theora?" because this link is what I was asking for. Thanks.

marrs2• 11 years ago

Does anyone care? https://people.xiph.org/~greg/video/ytcompare/comparison.htm...

TD-Linux13• 11 years ago

Yes, Theora is also royalty free. NETVC's goal is to surpass the quality per bit of existing codecs though, which Theora doesn't do.

ZeroGravitas3• 11 years ago

The key principle in Thor development is doing it in the IETF process. Theora is much more like VP8 and VP9 in approach (it was based on VP3). The technical approach, basically use fairly standard approaches while avoiding the patent landmines, is similar across all these projects, it's doing it in the IETF that's different for Thor and may aid acceptance in some quarters, and help avoid some patent issues.

nullc2• 11 years ago

Theora is a few generations old now-- it's available for anyone to use, but there is a lot of interest in improved performance.

Theora has also suffered somewhat from not being developed through an open process: On2 code-dropped one of the older propritary formats that they were uninterested in supporting anymore, and the Xiph side (which previously had just done audio) picked it up, formally specified it, and radically improved the implementation (of both the encoder and decoder) while retaining compatiblity.

Codecs thrive on network effect-- you might be happy to take some efficiency loss for improved licensing terms and patent assurance, but there are plenty of other who don't care (or at least won't care until the lawsuit arrives on their desk). It's not good enough to be very good in one dimension, to be wildly successful a codec needs to be great in many dimensions; a decade ago you could argue that Theora was in that space (and I did!), less so today.

... and wildy success is itself the ultimate performance objectve, not for the sake of the egos of the developers but because only though ubiquity do codecs stop being an issue that people trying to deliver great works have to worry about. 99.9% of the time an application developer is not thinking at all about what his TCP stack is doing, who made it, or what its settings are-- ultimately that kind of effortlessness needs to move up the stack in order for the world to move forward.

datashovel• 11 years ago

There should be efforts outside of large corporations dedicated to building these standards. Because in general even when large corporations promise free / open-source licensing they really only mean non-commercial licensing or "open with caveats". So they pretty much own the commercial rights.

I want open-source to subsidize a small team of engineers to create a completely open standard where no single entity owns it and everyone is free to branch / fork it.

electriclove• 11 years ago

Why not simply work with the VP9 project rather than starting a new effort? Per Wikipedia: "VP9 is an open and royalty free[3] video coding format being developed by Google."

Ono-Sendai2• 11 years ago

What I would like to see is a video codec that has a library implementation for reading and writing video in that format, that is cross-platform and relatively easy to build, like libjpeg or libpng does for images. I have tried to build VP9 on windows and it was a tedious and ultimately unfruitful process.

I don't really care about the compression ratios achieved, or speed of compression/decompression.

Something like motion JPEG would be good, if it was actually a proper standard (AFAICT it isn't).

fndrplayer133• 11 years ago

Motion JPEG isnt resilient. H264/H265/VP9, etc all build on some of the ideas that JPEG introduces, but introduce features that allow for the stream to be resilient to dropped packets or frames.

Its a cool idea, it just doesn't work in practice. Especially since a lot of these video standards are transmitted over UDP.

Ono-Sendai2• 11 years ago

Interesting. I would have thought it would be reasonably resilient though, due to its intra-frame nature. If you get lost in the stream you could scan forward to the next JPEG header.

yabun• 11 years ago

There really needs to be a change to patent law around independent derivation of a concept. At very least we need to look into generalised thicket busting laws. The current situation is fundamentally unscalable.

shmerl3• 11 years ago

So Daala will be fused with Thor like happened with CELT and SILK to create Opus? Does it make sense technically, or they are radically different?

Thor – A Project to Hammer Out a Royalty Free Video Codec

Related Stories