Avoid FIFA World Cup CDN Issues

https://blog.peer5.com/avoid-cdn-issues-at-peak/

by shacharz • 8 years ago

48 52 8 years ago

blog.peer5.com

lephty• 8 years ago

Is there anything that can be done from the user perspective to improve your streaming experience. Beyond bandwidth.

shacharz• 8 years ago

If you're ok with lower resolution then you can lower it, if the service allows it. Obviously remove anything that might use network.

yAnonymous• 8 years ago

Use a VPN from a location where you know the service has enough bandwidth.

inglor2• 8 years ago

Hey, I work at Peer5 (another Peer5 employee wrote this article). We typically measure video user experience in several ways:

- The amount of rebuffering the user is getting (basically, the less the user sees the loading wheel the better). - The bitrate the user has (are the users seeing the video in the highest possible quality?) - Whether or not there are any media errors. - The amount of time the video takes to load after a seek.

From the user experience - you don't really have control over these things - it's up for the broadcaster to set up a good service.

We've found (unsurprisingly) that services that are either paid or are national broadcasters offer better user experience than ones that are free and easily found online so my recommendation would be to spend a few dollars and get a solid provider.

(Also, the bandwidth you get now doesn't mean too much when the network is very congested - so it's worth checking how fast your network is during big peaks and considering a different ISP or a streaming provider that utilizes P2P)

sergiotapia2• 8 years ago

Life pro tip: Just use an ace stream and have zero connectivity issues.

https://www.reddit.com/r/soccerstreams/comments/82ac7j/acest...

My brother bought the MMA ticket to watch mcgregor vs mayweather and it was a horrible experience.

I booted up my laptop and ran an acestream, boom crystal clear high definition image with zero network issues.

untog• 8 years ago

Of course, those streams are not legal.

justherefortart• 8 years ago

I've paid for 4-5 different streaming services over the last few years trying to see matches. They inevitably don't work, which is the only reason I use streams I find online.

If some random dude with pr0n popups can make it work but Fox/NBC/ESPN/etc can't, tough shit.

oh_sigh• 8 years ago

Is it illegal be the streamee, the streamer, or both?

nikanj• 8 years ago

Ah, that explains the superior user experience.

Here’s hoping sports get the memo re: usability. Music piracy plummetted after Spotify arrived on the scene.

sergiotapia2• 8 years ago

Yep, be smart or take the single "warning letter" from your ISP.

parthdesai• 8 years ago

Use Soda player for ace streams if you are on Mac :)

isostatic7• 8 years ago

What amazes me is the lack of concern on latency for web streaming.

We use the internet (and IP in general) to stream video. At high bitrates (200mbit+) we aim for sub 100ms end to end, for compressed services we're happy with 500ms, maybe upto a second if it's something like Sydney to London over the internet.

I was in a control room a couple of weeks ago watching some football. There were two displays, one end was the feed from the stadium, one was the feed from the web streaming service.

There were cheers and then groans from the live end of the room. nearly a minute later someone on the web end started running up the field to score. Of course I knew at that point that it wouldn't be a goal, as not only did the people watching the live stream tell me, but twitter was abuzz.

1 minute end to end delivery latency is shocking for this type of program. Heck 10 seconds is bad enough.

Someone1234• 8 years ago

Interesting factoid, but some press have agreed to intentional latency as a security feature around certain events and dignitaries. The theory being that a sniper or drone operator cannot use the "live TV" footage to know their exact position.

In some cases international viewers may see the "live" footage before local ones.

Harvey-Specter• 8 years ago

The theory being that a sniper or drone operator cannot use the "live TV" footage to know their exact position.

How is this helpful in either of these cases? A sniper needs eyes on, a drone operator probably has live video from the drone. I don't see how a TV delay would have any effect.

jonknee2• 8 years ago

"Live" TV broadcasts are also on a delay, so I guess it's all relative. I had a friend who lived near enough to an NFL stadium that you could hear when something big happened and the TV delay made it impossible to enjoy a home game there.

isostatic7• 8 years ago

Very minor delay in the UK -- well under 2 seconds from pitch to TV on DTT in the case of the FA Cup. Sure enough to cause some grief when you can hear the crowd, but not enough to get notifications on twitter or whatever.

mv45• 8 years ago

You comment mentions two types of latency:

1. network latency, in milliseconds, that affects stream quality and stability

2. the delay (lag) between real-time capture and what the end user is seeing; this one is usually measured in seconds. A stream needs to be ingested, transcoded, sent from distribution servers to edge servers in each target region - with each step adding to the delay.

Minimizing the lag is very hard, because stripping all buffers (to reduce the delay) makes the stream very sensitive to network conditions (which reduces quality). With most commercial CDN providers you will get 5..10 seconds. It can be reduced to 2..3 sec if you know what you're doing.

edit: in case anyone is interested - in the second scenario, where we achieved 2..3 sec broadcast lag vs real-time, stream source (ingestion) was in the US, and the viewers were in mainland China. Network latency was over 600 msec. Wasn't easy!

isostatic7• 8 years ago

60 seconds in the case of iplayer for the fa cup. 1m20 in the case of BBC News channel right now. HLS tends to be packetised in something like 15 second chunks at the end of the process.

I know why there's a delay, I'm just amazed that people aren't concerned about it. The BBC used to offer multicast sources of live TV, which is a far more sensible solution, far more bandwidth efficient and allows end-to-end transmission in the satellite (or even less) range.

Wowza did a talk at demuxed last year about how to do "3 second latency end to end at scale", which I found amusing given that TV people have been doing sub millisecond latency at scale for nearly 100 years, so at least some people in the industry recognize the problem (which is mainly for sports events)

_trampeltier• 8 years ago

Already since over 20 years, the german TV chanel "RTL" is about 15 seconds later than all other TV chanels on live events like formula 1. But I have no idea why it is.

mv45• 8 years ago

There no technical reason for it to be that high. Perhaps I should reach out to them.

LunaSea• 8 years ago

Luxembourgish channel

Reply

bradleybuda• 8 years ago

In the US we've seen a half-dozen over-the-top TV providers launch in the past couple of years. They all seem to understand that live sports is their core feature, yet none of them tries to compete on latency. I regularly see two or more minutes from reading tweets about a goal or touchdown (from broadcast or cable viewers) until I see that TD live on Sling / PSVue / DirecTV NOW / YouTube TV (I've tried them all). Second-screening live sports is impossible with the OTT apps, and is very likely to drive me back to Comcast.

There's a real opportunity for a sports-oriented OTT company to compete on latency, a DVR that actually works, and expansive rights (I never have to guess if I have access to any sporting event).

Reply

mtberatwork• 8 years ago

Doesn't "second-screening" while watching a sports event interfere with the act of watching the sports event in the first place?

Reply

mayneack• 8 years ago

I stream a lot of sports from home. I would gladly trade delays for stream stability/quality. I frequently have to switch from a legal stream that I pay for to a more robust illegal stream.

It would be annoying if there were multiple devices nearby on different delays, but for me in my living room. I don't care if it's 30 seconds or 3 minutes. I've gotten spoiled by twitter feeds a few times, but it's not the end of the world.

Reply

btown• 8 years ago

Software for scaling up live-streaming CDN points of presence (POPs) is a pretty crazy domain. For on-demand video, you can think of a CDN as a cache, getting known-ahead-of-time chunks. But what about for live streaming? It's not feasible to stream frame-by-frame directly from your encoding backend to all the viewers of the World Cup, over something like RMTP - you'd want to use a CDN. So typically, you distribute meaty (multi-second) HLS segments as individual video files, or collections of files, to your CDN; once available, they then need to be requested by browsers/mobile clients as a whole segment, over HTTP(S). Works well with existing CDN infrastructure (provided they can handle the write volume and have big enough inbound pipes)... but the huge issue is that the length of the segment plus round-trips is a lower bound on effective latency. And when interactivity is required, multi-second delays can be horrible.

https://www.wowza.com/blog/hls-latency-sucks-but-heres-how-t... is a great writeup. Another overview of the problem, and a proposed solution, is in this excellent article by Twitter here:

https://medium.com/@periscopecode/introducing-lhls-media-str...

In HLS live streaming, for instance, the succession of media frames arriving from the broadcaster is normally aggregated into TS segments that are each a few seconds long. Only when a segment is complete can a URL for the segment be added to a live media playlist. The latency issue is that by the time a segment is completed, the first frame in the segment is as old as the segment duration... By using chunked transfer coding, on the other hand, the client can request the yet-to-be completed segment and begin receiving the segment’s frames as soon as the server receives them from the broadcaster.

And Twitch's followup challenge:

https://blog.twitch.tv/twitch-invites-you-to-take-on-the-icm...

This Grand Challenge is to call for signal-processing/machine-learning algorithms that can effectively estimate download bandwidth based on the noisy samples of chunked-based download throughput.

(IMO) If you're thinking that this is all rather silly, and that live video streaming is not something that should be done over HTTP in the first place... there are a lot of reasons why this is the case. All the CDN POPs are optimized for HTTP GET requests rather than stateful sessions, and Apple's smiting of Flash removed a lot of incentive for innovation on RTMP servers. The ironic thing is that Internet connectivity is fast/reliable enough nowadays that RTMP might have been able to escape its association with "buffering" spinners, and would provide a much lower-latency experience. Hopefully there's better standardization in the future as live video becomes more mainstream.

Reply

isostatic7• 8 years ago

BBC is doing UHD, with HDR and HLG, for the world cup, with the top stream 36Mbit/second [0]

There are a limited number of "spaces" available [1] -- I think it's upto 100gbit of output.

Unlike with the FA Cup (where the BBC did a uhd trial), the world cup will have a lot of games during the week, where people will be watching from the office (although probably not UHD). This will mean far higher loads on the distribution.

Fortunately England's only 3 games are either at the weekend or at 7PM. The second half of the tournament will really stress the UK internet though, with both World Cup and Wimbledon on during the working week.

[0] https://www.bbc.co.uk/rd/blog/2018-05-uhd_hdr_world_cup_2018 [1] http://www.bbc.co.uk/mediacentre/latestnews/2018/uhd-vr-worl...

Reply

helipad• 8 years ago

"Only" – A subtle dig ;)

Reply

CapacitorSet• 8 years ago

The P2P approach is interesting. Do you have data on how much bandwidth is saved at peak usage?

It might be especially interesting when many users share the same connection, effectively achieving broadcasting - the CDN pushes the data to one client, and it broadcasts it to the local network.

Reply

cflat2• 8 years ago

Don’t use webrtc for live video. It doesn’t scale with CDNs. Use dash/hls.

Reply

jwandborg• 8 years ago

Could you elaborate on why WebRTC wouldn't "scale with CDNs"?

Reply

inglor2• 8 years ago

WebRTC is stateful whereas dash/hls are HTTP based and stateless - caching with HTTP is easy and CDNs have a ton of infrastructure on the internet to support it.

Caching with WebRTC is very hard - even for a single second - since every connection is stateful.

Reply

whadar• 8 years ago

Peer5 uses webrtc datachannels and runs on top of existing dash/hls streams

Reply

cflat2• 8 years ago

still doesn't scale efficiently. http based streaming is the only reliable way to get economies of scale. The cpu cost per webrtc socket is high compared to a cache-hit for a static resource. Not to mention that WebRTC is way more cpu intensive client side compared to hls/dash which has kernel/hardware offloading.

Reply

petepete• 8 years ago

I watched the UCL final via YouTube this year and the Bet365 webapp kept telling me about goals half a minute before it happened on the screen. To be honest, if you're isolated the delay doesn't really matter.

Reply

lossolo• 8 years ago

I've started building P2P live adaptive (DASH) live video streaming using WebRTC with distributed rate control mechanism some time ago and I am planning to open source it. Basically using that you could build your own p2p global distributed live adaptive video streaming CDN (or use it on one server only). Adding new supporting server (to add additional bandwidth) in this solution is just as easy as spawning vm/server and launch binary. Distributed signaling server with GEO/region based peer distribution, full real time statistics on whole network health, analytics, network automatically adapts to bandwidth shortage (if for some reason network can't sustain itself) switching to lower bandwidth versions of the stream. Very easy to use, you would need to set one config file only, launch 2 binaries and add one JS file to source of your site and you are ready to go.

Anyone interested in this?

Reply

ctlaltdefeat• 8 years ago

Yes!

Reply

fasteddy760• 8 years ago

Yes! How do I keep updated with progress? Do you have a project hosted somewhere?

Avoid FIFA World Cup CDN Issues

Related Stories