How I Made Porn Video Streaming More Efficient with Python and C

http://www.toptal.com/python/how-i-made-porn-20x-more-efficient-with-python

by crm416 • 13 years ago

267 112 13 years ago

lmm• 13 years ago

It's easy to underestimate the power of switching to a better language by just doing it - guess at the syntax until it works, then refactor as you start to understand the language and its culture more. In fact I've found I learn faster this way than any other.

kbenson• 13 years ago

I love reading articles about technical issues and solutions in the Porn industry. It's like getting a peek inside a youtube scale company as they grow.

niggler4• 13 years ago

Are any of the porn sites larger than youtube?

mseebach• 13 years ago

The question isn't unambigiously answered, but there's a top of good stuff here (first link in the article):

http://www.extremetech.com/computing/123929-just-how-big-are...

"In short, porn sites cope with astronomical amounts of data. The only sites that really come close in term of raw bandwidth are YouTube or Hulu, but even then YouPorn is something like six times larger than Hulu."

magikbum• 13 years ago

I would rather learn if there was a porn site bigger than Netflix.. as Netflix currently uses around 30% of US bandwith [1]

1. http://www.pcmag.com/article2/0,2817,2395372,00.asp

rorrr2• 13 years ago

No way. One year ago Youtube users uploaded an hour of video every second.

christopheraden• 13 years ago

Disregarding the morality and ethics that are often center-stage, they are faced with really interesting technical hurdles. It's an old article, but I was fascinated to hear about some of the challenges they face as well: http://highscalability.com/blog/2012/4/2/youporn-targeting-2...

toptal6• 13 years ago

Hey this is awesome! Though the admin/moderator changed the title for some unknown reason from the title of the blogpost to their own.

vsh2• 13 years ago

Of course because of snobbery. NH visitors should be protected from words, which starts from "p" and ends with "orn".

niggler4• 13 years ago

"NH visitors should be protected from words, which starts from "p" and ends with "orn"."

popcorn? preworn?

edmond_dantes• 13 years ago

"Porno" sounds so much worse.

Finster• 13 years ago

Like peppercorn. Wendy Peppercorn.

niggler4• 13 years ago

trigger-happy, i guess. Given that "Porn" definitely implies a much larger scale than merely "video streaming", I think the title change does a disservice

jdc• 13 years ago

It was kinda link-baity and vague...

toptal6• 13 years ago

jdc... it's the name of the actual article...

pavel_lishin• 13 years ago

For what it's worth, the new title is more accurate and more likely to get me to click.

wodow2• 13 years ago

This happens quite a lot / has become standard on HN (since sometimes in the last year).

mezeek• 13 years ago

This title and the article's title don't quite match...

crm4162• 13 years ago

Submitted title matched the page, but it's been changed by the mods

xSwag• 13 years ago

>The aggregated bandwidth of the clusters was around 50 Gbps, from which they used around 10 Gbps while at peak load.

Now that is a lot a porn.

Also, the illustrations look really good! How did you make them?

OtherPlanet• 13 years ago

Very glad you like those illustrations, first we have created couple of the sketches on the paper and then recreated that in the Photoshop.

tarikozket• 13 years ago

What service do you use for flowcharts?

i5rider• 13 years ago

Great work. I worked with RTMP for some time and I recall the pain to reverse-engineer their protocol. One other comment, maybe the title should be more like: "How I improved a slow/inefficient RTMP video streaming service by 20x". Just a thought.

hackerboos• 13 years ago

Would he get the same CTR if he changed the title?

jallmann2• 13 years ago

Agreed that RTMP is an abomination that needs to be exorcised from the Earth as soon as possible. Unfortunately, it is probably here to stay until something like WebRTC gains critical mass.

It's not clear what the article means by "repackaging" a stream or "pointers" to tags (especially in the diagram that shows tag pointers being transported between users). While RTMP is cumbersome, shoving media data (tags) under a per-session protocol header is essentially the standard way of moving data from one session to another.

So I'm not really following this. Is this cutting out the RTMP entirely for receiving clients, and instead sending the FLV down via another transport, like HTTP or whatever? Or is it more of, "I wrote my own RTMP streaming server in C and Python", along with some implementation details which I'm not understanding? (Not that there's anything wrong with doing so. Options are limited in the streaming-server space.)

toptalgergely7• 13 years ago

Author of the original post here.

>It's not clear what the article means by "repackaging" a stream or "pointers" to tags (especially in the diagram that shows tag pointers being transported between users).

By repackaging I meant extracting the FLV tags which pretty much travel in the same format as in an FLV stream (.flv file) if memory serves well. Pointers to tags refer to the internal implementation. I took the FLV tags out of the RTMP stream, which resulted in an almost complete FLV stream. With the proper header prepended to it you could save it as a file and play it or stream it and play it. That's just what I did, created a header for every new user, and after that was sent I could just stream the FLV tags from a common buffer. The users had pointers that pointed in this buffer so after the header is sent it was true multicasting.

> So I'm not really following this. Is this cutting out the RTMP entirely for receiving clients, and instead sending the FLV down via another transport, like HTTP or whatever? Or is it more of, "I wrote my own RTMP streaming server in C and Python", along with some implementation details which I'm not understanding? (Not that there's anything wrong with doing so. Options are limited in the streaming- server space.)

Yes. From the source RTMP stream I extract the FLV tags, which I could use to multicast. Sending the same RTMP stuff to every user would not work, but I can easily send the FLV tag stream over HTTP if I send the crafted header first.

I hope that helped

jallmann2• 13 years ago

> I extract the FLV tags, which I could use to multicast

I assume you don't really mean IP multicast (which would, incidentally, be one approach to edge-origin mirroring for the popular marketing campaigns you mentioned, at least within a data center).

Anyway that makes sense, simply sending the FLV is clever. What's the method of delivery -- chunked HTTP? Does it play well with proxies?

danso2• 13 years ago

I sympathize with why the mods changed the title from "Porn" to "Video", but now you've made it so that HN users are unwittingly clicking into an article that talks about porn, in such a way that could be detected by a workplace firewall. That's not an ideal usability decision. Why not use good ol square brackets to maintain the integrity of the title:

How I Made [Porn] Video Streaming 20x More Efficient with Python and C

FiloSottile• 13 years ago

Or the other way around

How I Made Porn [Video Streaming] 20x More Efficient with Python

as technically the editor notes are bracketed.

obviouslygreen• 13 years ago

I think your sympathy is overapplied here... that's very clearly an important bit of information for anyone considering clicking on this link. Thanks for putting the warning up for those of us who would not otherwise have known thanks to a poor title edit.

coldtea5• 13 years ago

>but now you've made it so that HN users are unwittingly clicking into an article that talks about porn, in such a way that could be detected by a workplace firewall.

If you get heat about things like that in your workplace, then run, don't walk.

peterhajas• 13 years ago

Why do mods change titles of posts? This is frequently complained about on HN, and seems unfortunate.

philh• 13 years ago

Mostly they make things better, and nobody notices. Sometimes they make things worse, and people complain.

bitwize• 13 years ago

If your firewall flags up the word "porn", won't it also flag up HN for having "porn" in the title of a listed article?

hardik9882• 13 years ago

I appreciate the effort the mods take in curating titles, I really do - but please spare articles like this at least?

I clicked on this link at work (part-time at grad-school) and now I have a "how to run a pornographic website faster" link logged in my name.

brokenparser• 13 years ago

Said Mr. Hard Dick.

DufusM• 13 years ago

hardik is actually a common Indian name.

sc00ter• 13 years ago

Oh, please. The article uses the word porn exactly three times in the introductory paragraphs. Four if you count the title.

The comments here have double that, including your own use. If you were genuinely that paranoid about being "logged" for having visited a page that used the word porn (really?!) I doubt you'd be using it yourself.

hardik9882• 13 years ago

I apologize if it sounded that way, but I wasn't insinuating that I'm going to get into trouble.

I'm just saying that it could get someone into trouble.

As far as the "logging" goes, it had more to do with the word porn in the title of the page, because many content-filters just parse the title for blacklisted words.

jeremyx3• 13 years ago

What is the association with this site toptal.com? Has anyone here ever worked with them before?

toptal6• 13 years ago

Hey Jeremy, this is our engineering blog.

jeremyx3• 13 years ago

Thanks. This is the only article on the blog? I was actually intrigued by the site and was wondering if anyone on HN has used this service as a dev or a client....

ckdarby• 13 years ago

Nothing open source in this article which was the biggest drawback of me reading this.

toptalgergely7• 13 years ago

Unfortunately that's the case. I heard really good things about wowza these days. Not open source but a lot of things are open and flexible. Also back then red5 was also an ok alternative, not sure if that's still the case.

atechnerd• 13 years ago

More efficiently? For two minutes of video? I could see if you were trying to stream Zombieland or Spaceballs... but porn?!

I guess whatever inspires people to innovate is okay with me.

toptalgergely7• 13 years ago

Original writer here.

Actually this was a website which streamed online broadcasts live from people, like ustream does but with adult content. Some broadcasts could be hours long.

dangayle• 13 years ago

I got hit with a firewall and a "This event will be reported".

Thanks, HN Moderator.

jneal• 13 years ago

I know others already said this, but I clicked on this link from work and immediately became appalled when realizing it was about porn and quickly backed out. I'd gladly read this from the comfort of my own home and I'm sure the content itself is SFW but still the point is I was mislead. Can we please [mods] not change titles in cases like this?

Myrmornis3• 13 years ago

Don't understand. You can browse articles for leisure at work. So what was the problem with this article?

buster• 13 years ago

Strange how 3/4 of the comments are more concerned with the title link, instead of the actual content.

Very nice post, was an interesting read!

tekacs• 13 years ago

To be fair I clicked on the article and immediately came back to the comments expecting the absolutely inevitable sprawl of comments about the title edit. :P

So not strange, methinks. :P

proland• 13 years ago

If you're going to change the title, at least add NSFW to it...

specialp• 13 years ago

How is it NSFW? It is an article that is about video streaming on scale. Yes it does mention it was for a porn site, but it really does not have much to do with the article. If you work somewhere where this would be viewed as inappropriate I am not sure what can be viewed at that employer.

Myrmornis3• 13 years ago

There's no reason anyone should cater to the consequences of you working at a prudish and illiberal organization.

ConceitedCode• 13 years ago

Is there an open source project that accomplishes something similar?

bochi• 13 years ago

There is on open source a Nginx module that streams RTMP and creates HLS segments that looks promising: https://github.com/arut/nginx-rtmp-module

There are also red5 (http://www.red5.org/) and rtmpd (http://www.rtmpd.com/)

chopsueyar• 13 years ago

Is this similar to what Wowza (wowzamedia.com) does?

In terms of licensing, it is $55/mo/instance or $995 for a one-time license.

There are also EC2 instances that start at 15 cents per hour.

Much less expensive than FMS

mtrimpe• 13 years ago

You also have Red5 which is open source and rocks very much. It is just a -tad- less reliable than Wowza which you can't really sell to paying customers, but it's much more fun and customizable.

And to answer your question, Wowza is just a cheaper FMS which also supports other platforms next to RTMP. This article is basically taking the more performant HTTP download mechanism for static content (like YouTube uses) and then hacking it to put a live stream in it instead.

ericcholis• 13 years ago

The network graphics are pretty sharp, anybody familiar with where they came from? Or, are they custom?

coldtea5• 13 years ago

I'm astonished by some of the responses here, that could be summarised thusly:

>"I work on workplace, in a modern western society, not some theocratic backwater, that monitors my web activity and would frown if I visited an article with the word porn in it. This on 2013. I find this OK, and won't quit my job or raise hell protesting this degrading treatment, but would rather complain for HN titles".

In an age where people fight for LGBT rights, this is what the American workplace has come to?

sillysaurus• 13 years ago

It's not that simple when you have a family to provide for. The job you have might be the best job for many miles.

I don't agree with the decision not to protest, and I don't think HN should really care about catering to that demographic, but I understand their motive.

danso2• 13 years ago

Since when did workplaces become bastions of principles, rather than bureaucracies that have a business to run and have to, at some level, apply policies that suspend an employee's worktime freedom as part of the agreement that involves paying that employee's salary?

I work at a small company where no one would care. They also don't care if I spend time on Facebook, Twitter, etc. etc. But not everyone is in a company like that. Big legacy companies hire from the HN crowd too. Whether or not their policies are fair is not always a simplistic argument. However, what is as sure as rain is that trivial violations of those policies can be used as leverage to punish people, when the office politics get dirty and desperate.

It's best to let the worker -- the HN reader in this case -- make the decision whether he/she wants to pick that fight, rather than have them accidentally stumble into it.

coldtea5• 13 years ago

>Since when did workplaces become bastions of principles, rather than bureaucracies that have a business to run and have to, at some level, apply policies that suspend an employee's worktime freedom as part of the agreement that involves paying that employee's salary?

Since people let them get away with it. The "agreement that involves paying that employee's salary" does not meant they should get away with treating him less than a civilized society accept.

Signing a contract to work on some place doesn't give the employer any inherent rights over the employee, besides those that the society is willing to accept.

Hitting an employee was once tolerated. Not so anymore. One time not hiring or paying blacks less was allowed. Not so anymore (not explicitly at least). Child labour was allowed. Not anymore. Racial or sexist slurs were allowed. Not anymore. Lax safety at work was tolerated. Not anymore.

So it's not like there is some undeniable inherent right of an employeer to "have the employee pee in a cup" or to "check his Facebook profile". It's just that people haven't protested enough to make it into law that it's not his fucking business what the employee does at his own time.

>It's best to let the worker -- the HN reader in this case -- make the decision whether he/she wants to pick that fight, rather than have them accidentally stumble into it.

Sure, but at least some anger should be directed against businesses having those policies, not just on HN titles, as if the policies are OK.

untog• 13 years ago

Where exactly is the link between porn and LGBT rights?

livnev• 13 years ago

The overall message is stop worrying so much about the locations of other people's genitals.

coldtea5• 13 years ago

The link is that sex (even gay/lesbian etc sex) is not a sin, what people like sexually is not the business of their employeer, and if we can accept people doing parades for the right to fuck each other in the ass or similar, we sure should be able to accept people reading the word "porn" on a website at work.

That's the implicit link. The explicit link is that LGBT advocates and activists have frequently defended porn and freedom of sexual expression through it, too.

_delirium• 13 years ago

Unfortunately, the majority of jobs in the United States are run with this kind of degrading treatment. Many even require you to randomly pee into a cup on short notice, to make sure you didn't do any drugs recently. Some of them also monitor your Facebook accounts (as far as permissions allow) to see what you're up to there. A few require you to hand over your Facebook passwords (!) to the boss to make that easier. The corporate world is weird and scary, but not always easy to avoid.

SilasX2• 13 years ago

>A few require you to hand over your Facebook passwords (!) to the boss

That was a poorly-source, probably-made-up story that now one could verify yet quickly became accepted truth. (Unless it's meant to refer to cases involving heavy security clearances, in which cases it's ho-hum)

Myrmornis3• 13 years ago

It's easy to avoid.

belorn• 13 years ago

> In an age where people fight for LGBT rights, this is what the American workplace has come to?

Yes. With a lack of privacy expectations in 2013, any visited website is permanent added to the record. Even with no paranoia, it does add an unnecessary risk to ones career.

nacho2sweet• 13 years ago

LOL at everyone who works at places that fear that the company firewall saw them read an article that had the word "Porn" in it.

Hopefully you get in trouble and fired, it will help your life in the long run.

serginho• 13 years ago

I've never expected that the story would be about porn.

ianhawes• 13 years ago

If anyone is interested in an alternative to the usual RTMP servers (FMS, Red5, and Wowza), I highly recommend EvoStream (http://www.evostream.com). Compared to the alternatives, EvoStream is much more efficient. I believe TinyChat published a whitepaper discussing their transition from Red5 to EvoStream, which resulted in a decline in the number of required servers.

EvoStream is a highly scalable streaming media server written in C++ based off of the open source RTMPD (http://www.rtmpd.com). The commercial company, also called EvoStream, is a relatively new startup and they do great custom work for those not familiar with streaming media/RTMP.

coherentpony• 13 years ago

Congratulations to whoever messed with the title; I clicked this at work.

mariuolo• 13 years ago

Why is the type of content relevant? Perhaps there's something peculiar about porn users?

protomyth• 13 years ago

Because, like it or not, the porn industry dealt with a lot of the technical issues of selling video, pictures, and video streams on the internet before anyone else. They needed it to be profitable / have actual revenue.

Killswitch• 13 years ago

Nothing. I can talk about my work on a blog or to a technical person without mentioning once that it's porn related and I've been a programmer in the porn industry for over 10 years. They used it as a way to gain viewers.

latchkey5• 13 years ago

As someone who built the infrastructure for serving porn for Kink.com, I'd say that this was a total waste of time. Spend the money on a third party CDN and serve from there.

swalsh• 13 years ago

I've always been curious, how is working for a porn company affected your social life? does it at all?

latchkey5• 13 years ago

I worked behind the camera, not in front of it. Sorry, I'm not interested in satisfying your curiosities.

iso8859-1• 13 years ago

With these amounts of bandwidth, does that still make sense? Sounds like this was a one-man job that worked.

latchkey5• 13 years ago

I'm not really sure what you mean, but absolutely using a CDN product is the way to go when you want to sell your content on a global scale. Otherwise, you are going to deal with having to build / maintain your own and the cost of doing that is far higher than just letting other experts deal with the issues that surround it.

kibibu2• 13 years ago

The article suggests that this is for live streaming shows. Would a CDN based approach work in this use case?

latchkey5• 13 years ago

Take a look at KinkLive.com. We were the first porn company to do live streaming in HD using a CDN (Bitgravity), all paid for, by the minute, with a micro currency system (kinks) that we built.