Show HN: Boltstream – Self-hosted full end-to-end live video streaming platform

by benwilber0 • 6 years ago

510 90 6 years ago

Have you performed any load testing or measured performance and scalability?

gruez• 6 years ago

Wouldn't it be mostly IO bound? A 10 Mbit/s stream would equal to 100 viewers on a 1Gbit connection, which you can probably handle with even a 1-2 core VPS. The primary consideration would be your transit/server provider and how well peered they are in relation to your viewers.

corobo2• 6 years ago

If you're chucking it out as you get it aye, but chances are you're going to need to do a bit of transcoding aren't you?

VWWHFSfQ• 6 years ago

This is really great. I've seen so many of these "self-hosted live video streaming" sites show up here and this is the first one that show the actual backend code. Usually they're just web frontends to some video streaming website. Questions:

-[0] How widely deployed is this stack (specifically nginx rtmp/vod) around the live-streaming industry?

-[1] You listed under the Feature that it support "Pay-per-view" and "Pay-per-minute" -- how exactly is that enforced if the playback is just regular HLS? (Are you checking every single request for a .TS segment in Django?)

-[2] VOD/Recording is really cool but has enormous storage costs. How does BoltStream handle the long-term storage of streams?

Edit: re [1]

It looks like there's a somewhat sophisticated mechanism [here](https://github.com/benwilber/boltstream/blob/master/ansible/...) that's just checking the requests for the encryption key instead of every single request for the MPEG-TS segment.

This is awesome though. Thanks for sharing it!

spenczar56• 6 years ago

[0] RTMP is common; `nginx-rtmp` is not common in industry to my knowledge. RTMP ingest is a pretty different task from HTTP reverse proxying. Wowza used to be the common software that was used. Twitch moved off of Wowza and wrote their own ingest stack when I worked there.

ta123523567• 6 years ago

Facebook live (https://engineering.fb.com/2015/12/03/ios/under-the-hood-bro...) and several others use nginx-rtmp. It's fairly common in china as well from what I've seen.

RTMP is still the best protocol for publishing live streams IMO, and in my experience http-flv blows HLS out of the water for < 1 sec latency. The protocol is old but at least you aren't 15 seconds behind the action.

dayjah• 6 years ago

Serendipity is great! I was literally thinking about this today with a view to improving the personal training space.

I’ve a few friends who are personal trainers, they all complain about the “zoom centric” solution where by as a client you dial in to a zoom call, and have to move a phone, iPad, whatever, around a bunch so the PT can see you.

My mates have suggested that what they really want is to be able to select from a range of views while a client performs some maneuver.

While this doesn’t solve the upstart cost (multiple cameras, bandwidth, etc), it’s cool that it does solve the “hard part” of working out how to stream to some common end point that allows the viewer (the PT) to select the best view.

keyle3• 6 years ago

It's a chicken and egg problem where there is not much point in building your own whizbang streaming platform and have no one come to it.

Twitch is getting increasingly ad-nnoying and you can _ad_ barely _ad_ watch _ad_ it _ad_ an _ad_ ymore.

That said that's still where you get the viewers and that's what people will pick. The ad-revenue sharing being the carrot here which will probably mean it's going to be successful for a lot longer.

reificator2• 6 years ago

I run an adblocker but I'm sympathetic to Twitch's plight here.

Or at least I was until they started interrupting streams in the middle of the action with no human input to show ads.

When the broadcaster clicks the ad button, it's a good time to run ads. They're probably taking a break anyway, there's nothing critical or even interesting happening, so watching some ads is fine.

But randomly interrupting in the middle of a conversation or right as the good part hits to show an ad shows that the Twitch management no longer understands or gives a shit about their platform.

codetrotter• 6 years ago

They probably found that people streaming were not clicking the button often enough.

spenczar56• 6 years ago

That's been a feature since the Justin.tv days. Some broadcasters choose to automatically run an ad every 30 minutes, for example. Twitch's broadcaster UI supports this, and has for a very very long time.

keyle3• 6 years ago

mid stream randomly, you get thrown 30+ seconds of ads, great if you were having a conversation with the host.

then if the stream breaks, refresh, new set of ads.

pop the popout player, ads.

It's totally rooted. The suits run the show and they're milking the cow. The only reason they're going so hard at it, I speculate, is that they're being hit by big DMCA fees.

I can't otherwise explain with logic why they'd run ads literally on DJ streams with thousands of viewers.

HHC-Hunter• 6 years ago

This is really needed right now, thank you for making this!

More self hosted solutions people! Less centralized things.

killingtime742• 6 years ago

Great project!

killingtime742• 6 years ago

Really great project to decentralise video

2Gkashmiri3• 6 years ago

Peertube is getting live streaming capability.... actually you can check it right now on https://live.canard.tube/videos/watch/5e7b916e-c5b5-4996-a97...

The p2p part of it is the real nice deal and these guys are actually making something good here.

InitEnabler• 6 years ago

Yea, I'm looking forward to that feature. If they can implement that with a peer to peer stream that would be great.

2Gkashmiri3• 6 years ago

P2p is already a thing and is working much better than expected. https://framacolibri.org/uploads/default/original/2X/9/9530f...

Here is some peer watching a stream (in french) and look at the stats.

eli_front• 6 years ago

How secure do you think Twitch (Amazon), Facebook Gaming (Facebook), and Youtube Live (Google) are at the top of the streaming world? We’ve recently seen Mixer (Microsoft) fall, and it feels like the creators are the only thing keeping these platforms great, so what’s stopping a new streaming platform from rising above all the others other than the creator’s fear of losing stability? Why couldn’t a new streaming service with a great user acquisition strategy fully take over? This industry doesn’t feel stable at the moment, but when has the video content industry been stable.

lovehashbrowns3• 6 years ago

There are so many things. First, the network effect. Going from an established platform to a new one means you lose a lot of viewers that aren't going to switch and the potential to gain new ones since the other platform is new.

Secondly, well-established payment methods such as Twitch Prime which provide a lot of revenue for streamers on Twitch.

Third, existing contracts with Twitch.

Fourth, streaming platforms are incredibly expensive to create, maintain, and promote. They will not be profitable for a long, long time, on top of requiring mountains of cash to start.

And finally, users "get used" to a specific platform, e.g. Twitch chat, and it's difficult to break these emotional attachments.

holler2• 6 years ago

Fourth, streaming platforms are incredibly expensive to create, maintain, and promote. They will not be profitable for a long, long time, on top of requiring mountains of cash to start.

I agree with that and unless you have founders who possess the required expertise + motivation, it will be expensive.

And finally, users "get used" to a specific platform, e.g. Twitch chat, and it's difficult to break these emotional attachments.

Yes and no, I think if you build a quality product, people will try it, and if it works and is delightful, they'll come back. Look at TikTok.

skocznymroczny• 6 years ago

As a gamer, I've never even heard of Mixer, and I don't think I am familiar with Facebook Gaming. I am familiar with Twitch and Youtube live, although I don't really watch livestreams, only "Best of" compilations on Youtube. The community around livestream chats is turning me off, with it's endless spam of "pog" and other memes.

michaelmarkell2• 6 years ago

Maybe I can latch on to this thread to ask a question — has anybody here had any experience with publishing streams from the browser directly, rather than using something like OBS?

Current thinking is I could use the MediaRecorder API to capture browser content and then push it to a server for broadcast with a tool like Boltstream. But, I’ve heard some people say it’s a bad idea to try this, and to stick with non-browser options.

contrahax• 6 years ago

Streamyard does this, works pretty well but less customization available right now.

michaelmarkell2• 6 years ago

Thanks for the tip!

Ceezy• 6 years ago

Nice!

sim_card_map• 6 years ago

Strange to see a python backend for such a performance critical application.

jyap• 6 years ago

It doesn’t do anything performance critical.

nginx-rtmp-module Handles the media streaming.

Use something else like ffmpeg to stream.

spenczar56• 6 years ago

Cool! Some thoughts from a former Twitch engineer:

- Probably the hardest part of running these things is managing outbound bandwidth costs. You'll need to either limit inbound bitrate or transcode video down to a manageable rate, or else you'll quickly spend a lot of money on shipping 4k video for people.

- Right now, your nginx hosts both do ingest and playback, if I understand it right. You might want to separate the two. It makes maintenance easier, and it lets you scale much better - right now, a single stream probably maxes out on viewership based on the CPU capacity of the single nginx host that is ingesting the stream, transcoding it, and delivering it. If you have multiple nginx hosts which could deliver the already-transcoded stream, you could scale much better.

- Please don't keep using RTMP. RTMP is so stateful that it's pretty hard to manage, it doesn't have a spec, it doesn't have implementations on many devices, and its security is, uh, _weak_. Big players are forced to keep using it because telling their broadcasters to change is hard, but you don't have that problem. You might consider accepting an alternative modern protocol.

- You'll almost certainly need admin and moderation tools soon. Expect lots of pirate streams, as well as some horrific content. You can't run a live streaming platform without admin tools.

- Beware DDoS attacks. This setup looks very, very, very easy to take down, as-is...

toomuchtodo• 6 years ago

Please don't keep using RTMP. ... You might consider accepting an alternative modern protocol.

What would you recommend as an alternative modern protocol to RTMP for shipping video between a broadcast endpoint and transcoding/distribution infra?

grogenaut• 6 years ago

Spenczar might know better but SRT seems very interesting for modern protocols. RTMP is very old as a protocol.

spenczar56• 6 years ago

Well, to be fair, there isn't a clear winner.

Maybe SRT, but it doesn't have much platform support itself, and its spec is pretty strange. I get weird Adobe flashbacks about it, since it seems so heavily pushed by Haivision, and it's really made for a different use case - it's made for big broadcasters like ESPN and TV networks, not for web startups.

The current motion seems to be towards hacking WebRTC to accommodate broadcast ingest. But WebRTC is really huge and complicated and you only want a tiny sliver of it for this use case. Pretty much all implementations are buggy because its such a gigantic kitchen-sink design.

There's also RIST (for those who love working groups) and Zixi (for those who love closed source protocols?).

I honestly think that something like HLS could do pretty well (broadcaster slices video into 1-second TS segments and just ships em with HTTP POST), but it's not something I ever saw anyone do. Definitely doesn't meet the TV networks' needs, but its really easy to scale and straightforward to secure, which is maybe more important.

gizah• 6 years ago

(disclaimer: I'm participating in the libRIST development) RIST! It's a well thought out protocol without technical debt (which SRT has), supports multicast (if you need it), null packet deletion, encryption via PSK or DTLS and multiplexing (multiplexing support in libRIST is not fully there yet, it's on the roadmap together with DTLS). Next iteration of the RIST protocol will support "backpressure" on your encoder, so if you have too much packet loss your encoding can scale it's video bitrate down, while together with null packet deletion you'll be able to keep your bitrate perfectly CBR.

iso16315• 6 years ago

RTP, if you need error correction use RIST simple profile.

It's really simple and you can see exactly what's happening on the network.

nowahe2• 6 years ago

Having worked with AWS IVS, which is using the same infrastructure design as Twitch (or the other way around), it only accepts RTMP (well RTMPS to be fair) as an input.

To me it would seem that the industry is going the opposite way, and doubling down on RTMP. An RTMP connector is planned on the Chime SDK as well (a framework & infrastructure based on Chime to create your own custom meetings), to be able to stream the meeting directly to ingest service (like IVS/Elemental). On the AWS side at least, there doesn't seem to be a plan to migrate from RTMP, as it seems that services are being launched that only supports it.

On a sidenote, the official AWS recommended way to stream a Chime SDK meeting to IVS, is to use this[0] docker container, acting as a bridge between WebRTC & RTMP. I find the hackiness of it amazing, as it's mostly just an X11 framebuffer, an instance of Firefox loading the webview for the meeting, and ffmpeg capturing the framebuffer and sending it as an RTMP stream to the configured endpoint.

Also, AWS IVS as a streaming platform is really impressive. It's a fully managed service, and out of the box you get around 4-5s of latency between the source and the player, with an upper limit of 1080p60 @8.5Mbps. It will also downsample the stream to 720p, 480p, 360p & 160p.

[0] https://github.com/aws-samples/amazon-chime-meeting-broadcas...

jiofih• 6 years ago

Is 4-5s latency considered good? It’s an awful experience for video chat.

captn3m0• 6 years ago

to use this[0] docker container

In a similar vein, Jitsi's video recording solution is pretty much the same thing[0].

It works by launching a Chrome instance rendered in a virtual framebuffer and capturing and encoding the output with ffmpeg. It is intended to be run on a separate machine (or a VM), with no other applications using the display or audio devices. Only one recording at a time is supported on a single jibri.

We are severely lacking stand-alone webRTC implementations - the best implementation is in the Chrome codebase.

[0] https://github.com/jitsi/jibri

Torkel• 6 years ago

How do you handle hevc/h265 and resolutions above 4K?

It’s not in the rtmp spec, so not supported by either ffmpeg or gstreamer. But chinese companies ship cameras that hack it and run rtmp for it anyway, leading users to peskering me about our sw being broken :/

iso16315• 6 years ago

out of the box you get around 4-5s of latency between the source and the player

In the world cup in 2018 I was watching an RTP feed direct from the IBC in Moscow on my desktop in VLC. The goal went in, and strangely I recognised it was a goal for England (I don't really do football). Window was open, and I cheered.

A few seconds later the neighbours cheered - they were watching it on TV.

I remember watching a 4K FA Cup final too in parallel to an off air TV feed. Watched the goal go in on the TV. By the time it had gone in on the 4K feed I'd actually forgotten it was a goal, so I guess that wasn't too bad.

That's a problem with streaming. If you're watching a popular live event like a big football match, and your neighbours are too, you need to be receiving it at the same time to avoid "spoilers".

sneak2• 6 years ago

Probably the hardest part of running these things is managing outbound bandwidth costs.

This might be a good time to mention that Hetzner (where I host) has a lot of unmetered gigabit options. I'm not affiliated, just a happy (long term) customer.

Sadly Germany has some laws that censor certain types of harmless content, so that's one downside, but that's not Hetzner's fault, just something to consider depending on what content you intend to host.

dewey• 6 years ago

Sadly Germany has some laws that censor certain types of harmless content

What is this harmless content that's censored?

nielsole• 6 years ago

You could use their Finnish dc. (no idea if Finnish laws are any better in this regard)

chaz6• 6 years ago

I can attest to Hetzner. I pay €35 per month for a dedicated server, and one of the things I run is a (non-exit) Tor relay, which consumes 3.5TiB per day on average.

supermatt• 6 years ago

Any specifics on DoS? What sort of strategies to prevent a malicious user pushing multiple streams to DoS encoders, etc?

ChrisMarshallNY• 6 years ago

I don’t like RT[M|S]P, but I have not seen an acceptable alternative for realtime streaming.

The main reason I don’t like RTMP, is because it isn’t supported at the OS level in Apple systems (the platform I program for).

HLS doesn’t come close to RTMP. It’s designed for movie streaming, and doesn’t support realtime streaming. I write video surveillance software, and the HLS lag is unacceptable.

zweizeichen• 6 years ago

Some great points! I built a project similar to this (think online lectures but it never went anywhere) in spring when the pandemic hit. I highly recommend building a platform like this if you have some spare time. It involves lots of interesting engineering/architecture challenges all over the stack. IMHO, from a technical viewpoint, it's mostly integration work, as all the really hard parts were already done by excellent 3rd party tools and libraries. You'll probably have some kind of RTMP bridge (nginx [1]), wrap ffmpeg for transcoding and play it back using something like Shaka player [2] where the segments are being served by some kind of caching reverse proxy. It took me a few weeks to get a prototype running inside a K8S cluster where oAuth-ed users could publish RTMPS streams which could be watched by other authed users via HLS in a somewhat scalable way. It was surprisingly easy to build this using Elixir/Phoenix LiveView. Some thoughts on your comment:

- Probably the hardest part of running these things is managing outbound bandwidth costs.

This. As others noted, you may be getting around this by using something like Hetzner's/Scaleway's/OVH's offerings. However, I think they won't be too happy if you really use that much bandwidth all the time. You can probably forget using IaaS of one of the bigger cloud platforms unless you negotiated some special terms.

- You'll almost certainly need admin and moderation tools soon.

That's one of the main reasons why I did offer a public instance. It's probably best to host a platform like this in a non-public context such as members of a single org. Just look at what happened to Firefox Send...

- Please don't keep using RTMP [...] and its security is, uh, _weak_.

Yes it is a bit of a pain to work with but AFAIK you can wrap RTMP inside a TLS session which is supported by OBS. I think I just exposed a stunnel instance back then which forwarded it to a nginx instance which handled auth/auth and forwarding to the transcoding backend. This way you won't leak any streaming keys or video data. Please correct me if I'm wrong. If you have any additional pointers regarding RTMP security, I would be highly interested!

Also, as others pointed out, SRT may be just around the corner. I think we're in for some interesting times as LL-DASH/CMAF and similar technologies are more widely supported. Additionally, there are interesting projects like [3] going the WebRTC route for delivering low latency streams.

[1] https://github.com/arut/nginx-rtmp-module

[2] https://github.com/google/shaka-player

[3] https://www.ovenmediaengine.com

andrenotgiant• 6 years ago

FWIW, the OP repo uses a Terraform config to deploy the server on a 4GBRAM server on DigitalOcean.[1] Which costs $20/mon and has 4TB/mon bandwidth allowance.[2]

To give a sense of what a 4TB bandwidth could cover: 320 people watching a 3hr stream at 1080p or 70mb per minute.

[1] https://github.com/benwilber/boltstream/blob/master/terrafor... [2] https://www.digitalocean.com/community/tools/bandwidth?activ...

stormbrew2• 6 years ago

"Please don't keep using RTMP."

So, this has been talked about a bunch already, but I'm involved in a small streaming group that occasionally puts on (online, not in person) marathons and we've been looking for something to use that's better than rtmp for a while for ingest from participants and we really haven't found anything good. At least nothing that's viable to set up with open source software of your own.

We just piloted SRT (having about 35 different streamers stream to us with it from OBS into srt-live-server, then into OBS with a media source), which has been mentioned a bunch here, and it's been a mixed bag and I don't know if we'll stick with it.

The big pro is that it doesn't fall behind like the usual RTMP chain (OBS->nginx-rtmp-flv->OBS media source or web source with flvjs) tends to when connection quality degrades.

I don't think this is necessarily an inherent characteristic of rtmp really, but the open software in this chain tends to just grow bigger and bigger buffers when the connection hitches, leading to progressively worse latency you have to 'fix' by resetting the stream. Twitch and other big ingesters seem to avoid this somehow, and I suspect ingesting directly into HLS or similar avoids it, but low latency HLS is still essentially unavailable in open source packaging so is not viable for our use.

On the downside, though, SRT requires somewhat manual tweaking of latency and buffering parameters to work properly and our experience has been that, while the stream never falls behind, hitches outside the configured parameters produce a particularly ghastly kind of artifacting we've called "streakies" where entire segments of the feed will just turn weird bands of colours. It's possible this is actually a problem in the software, as the bits needed to put this together (particular srt-live-server) seem pretty immature yet.

It seems like the future for this kind of use case is likely to wind up being webrtc, but as of now getting webrtc to play nice with OBS is kind of painful, usually just involving using external software that captures a popped out preview display from OBS. It's a little less than ideal. I think it'll be our next thing to try though, and obs.ninja looks kind of promising as a platform to help with doing it.

simplecto• 6 years ago

This is very interesting. I setup a hacky nginx-rtmp thing to mimic some aspects of restream.io, as well as running a 24x7 live stream on a side project. It does work, but the hand-editing of configs means it does not drive easily.

This space is starting to heat up, and I would like to take the best open-source projects as well as the paid versions out there (dacast, wowza, etc) and do a full-on comparison.

I'm in the market for this software (and I'm not shy of paying).

shmerl• 6 years ago

Playback via standard HLS

Why HLS instead of DASH + MSE? Isn't HLS Apple only?

oplav• 6 years ago

Safari supports HLS natively, but hls.js is a widely used library that support HLS via MSE.

https://github.com/video-dev/hls.js/

echelon• 6 years ago

I have some questions for people in this field.

Is there a good way to broadcast to Twitch, YouTube, and Reddit simultaneously? Is it a bad idea to target multiple communities?

How expensive is it to run your own single-tenant stream on your own website? Is that a bad idea?

Are users revolting on Twitch against the ads and RIAA take downs? Or is it unlikely to ever be unseated?

corobo2• 6 years ago

Is there a good way to broadcast to Twitch, YouTube, and Reddit simultaneously?

Yes (disc: I dunno about reddit, do they have livestreams? haha I had no idea) restream.io is one service that does this, or you can configure your own nginx-rtmp

Is it a bad idea to target multiple communities?

Probably, the way you're suggesting. Find your niche and stream on the site they use. You'll look like you're talking to nobody most of the time unless your users gather in one place. Viewers (ok, it's me) don't like it when you answer a question from someone and can't see what the question was. At least factor that in if you go down this path - re-state the question

How expensive is it to run your own single-tenant stream on your own website?

Almost zero because nobody will watch it

Is that a bad idea?

Not really but it might be a waste of time

Are users revolting on Twitch against the ads and RIAA take downs?

Nah everyone else has forgotten and moved on. At the least folks don't care enough for there to be an uproar about it

Or is it unlikely to ever be unseated?

Twitch unseated? On the heading they're on now, probably. Literally all they need to do is stop ads and push subs and they'll be locked in for generations. A single sub is worth like 2500 ad views and people gift them in 5s, 10s, 20s like it's going out of style.. what are they even doing with this ad focus? The only ads I even see are ads for Amazon Prime shows I've already watched anyway! Who's even paying for these??

Wouter33• 6 years ago

Lot's of people are using restream.io. Good solution to stream to multiple places at the same time.

Doing you own single-tenant stream is hard to manage and scale in general.

reggieband• 6 years ago

How expensive is it to run your own single-tenant stream on your own website?

This is a function based on the number of viewers. The ingest and transcoding/transmuxing can be thought of as a fixed cost (and in many cases is free) and those operational expenses become rounding errors once the number of viewers surpasses a certain size. The real cost is egress (bandwidth leaving edge servers) and to a much lesser extent mid-gress (the cost of moving bytes between servers, e.g. from an origin to regional distribution server).

Is that a bad idea?

Almost certainly a bad idea from a financial perspective. You have a bad bargain here. Either you have so few viewers that the bandwidth is manageable (i.e. no revenue) or so many viewers that there is literally no way for you to afford the bandwidth. If there is a goldilocks zone in the middle it is probably a tiny sliver. You have to have the perfect number of viewers and revenue with very little margin for error.

I remember one popular Twitch streamer (sodapoppin) was doing a QA with viewers where he addressed the question. He had hired some people to figure out the costs of setting up his own streaming server and it didn't work out financially for him. Other streamers have investigated it as well (Destiny?) and came to the same conclusion. If those top streamers can't make it work financially, I doubt anyone can to be honest.

7357• 6 years ago

Thank you.

TrevizeGP• 6 years ago

Slightly connected: Does anyone found a good way to distribute HLS segments to multiple servers in almost real time? Let's say you have a setup with Nginx managing input RTMP and producing HLS segments in a local directory and you want those distributed to different "edge" servers. I can think of using inotify + rsync or pushing into an object store and let in handle replication (not sure if it would be fast enough?).

For various reasons (and for now) I would like to skip pushing them into S3 and use Cloudflare/Cloudfront.

OpuRahman• 6 years ago

You'll almost certainly need admin and moderation tools soon. Expect lots of pirate streams, as well as some horrific content. You can't run a live streaming platform without admin tools.

jaequery• 6 years ago

Anyone ever tried an app BIGO live? I always wondered how their videos can load and play so quickly, almost milliseconds in most cases.

Show HN: Boltstream – Self-hosted full end-to-end live video streaming platform

Related Stories