HN user
... guaranteed minimal latencies. You could now shave off another 20-100ms of latency for your FaceTime calls...
I already do this. But I cheat - I use a good router (OpenWrt One) that has built-in controls for Bufferbloat. See [How OpenWrt Vanquishes Bufferbloat](https://forum.openwrt.org/t/how-openwrt-vanquishes-bufferblo...)
The name "bufferbloat" is falling out of favor. First off, it has a horrible sound... And it doesn't really give an intuitive sense of what's wrong.
Referring to "latency" or (my favorite) "responsiveness" is better.
And I was encouraged to see this recent ZDNet article that mentioned the "ping rates" of 600-1000 msec, and notes that these would cause videoconferencing or gaming to be unusable.
https://www.zdnet.com/home-and-office/networking/i-tried-del...
And as @giuliomagnifico points out, you can fix it yourself if you get the right router.
What do you think of PRQL ("prequel")?
I just stopped updating the controller software (none of their gear is external-facing, and IIRC it's only needed for configuration/management) because cloud login is an absolute dealbreaker for me.
Yeah. Updates used to be a nightmare. I had to worry about Windows updates, Java updates, and of course Unifi updates.
Then I started to use the Docker container at https://github.com/jacobalberty/unifi-docker No worries about cloud login...
I have 21 APs all controlled by the container on a Raspberry Pi 4. It's not even breaking a sweat. When I want to upgrade the Unifi application, I stop the container, and re-run the command to use the newer Unifi version. Three minutes later, it's back on the air.
Interestingly, a significant portion of the recent WWDC presentation talked about decreasing buffering in server-side systems. https://developer.apple.com/videos/play/wwdc2022/10078/
As another poster noted, the Feds funded a lot of the cost of electrification. There is starting to be some action on the internet front. But...
Here are the economics for a fiber optics deployment in 2021 for a rural town of ~800 premises, where about 60% got service. They charge customers about $100/month for symmetric 75/75mbps internet plus phone.
It costs about $40K per mile to run the fiber down the road on existing utility poles. It costs between $2K and $4K to run the drop from the utility pole to the home.
Those two numbers (miles of road, number of premises) get you in the ballpark of the cost of deployment in a town/county. More premises per mile obviously decreases the cost per premise - that's why utilities prefer dense areas (cities).
@LinuxBender is on the right track. Bufferbloat can cause this kind of lag. (There may be other causes, but first thing to do is check for bufferbloat.) To do this use:
http://dslreports.com/speedtest
This measures the latency/lag while data is being transferred. When it's complete, click "Results + Share" and you'll see a report. Send us the link of that page so we can see your results.
Your router is holding ("buffering") packets in the hopes that they can be sent soon. Your measurements indicate that the router is "bloated", holding about five seconds (5,000 ms) worth of data.
This gives the sending TCP algorithm the wrong impression. It's waiting to hear about a dropped packet to indicate that there's congestion. When your router holds on to those packets (instead of dropping them), the TCP algorithm doesn't get any feedback, so it keeps shoveling data into the connection.
This leads to the bad state you're seeing. And that's where the advice on "What can I do about Bufferbloat?" comes in.
There's no benefit to having more than one packet buffered by the router. (Hanging on to more than one packet per connection only causes the latency/lag you're seeing.)
There are routers that actually check the time the router has held packets. If packets have been queued for "too long", the router discards them immediately, giving the vital feedback to the sending TCP. Those routers use the technique known as SQM (Smart Queue Management) and the fq_codel, cake, PIE algorithms to keep the queues within the router short - typically less than 5 msec.
To solve your problem, investigate getting a router that implements one of those SQM algorithms. They're listed on the "What can I do..." page. I am a fan of OpenWrt (use it at home), but have installed a bunch of IQrouters and Ubuquiti devices for friends.
Yes. Check for bufferbloat using http://DSLReports.com/speedtest
If you get a grade below "B", then your home router is likely bufferbloated. Then read "What can I do about Bufferbloat?" at https://www.bufferbloat.net/projects/bloat/wiki/What_can_I_d...
Please report back to the list so we can refine our knowledge. (It's like Covid-19 testing. We cannot know the incidence of infection if we don't test lots and lots of people to find a true rate...) Thanks.
You wrote:
Why should the video bitrate be dropped because some second rate ISPs are oversubscribing super badly? Isn't all of this totally against the principles of net neutrality?
No. This is entirely separate from net neutrality. That principle says, "No ISP should discriminate on the basis of content."
In the case of an oversubscribed ISP, the "magic of the markets" that so many lobbyists talk about is that you can simply switch to a new provider. Oh, you only have one provider? Tough cookies.
Yes, we need to emphasize this point.
Netflix (and Youtube and Hulu and all the other big content providers) pay big bucks (millions per month) to get the content from their servers to the internet backbone. Their contracts with their ISPs don't specify where the traffic goes. They just buy lots of 10 gpbs links, and send the data off. They're paying their share.
Your contract with your ISP doesn't distinguish between getting traffic from a mom&pop website or Netflix. It just said they would deliver the rated speed (7 mpbs or 10mbps, or 100mbs) of data to you for your fixed monthly fee.
When the ISPs found out that people were actually using that much data (that YOU PAID FOR), they found they had underprovisioned their network, and couldn't support the load. ISPs started saying, "Netflix is using too much", because, really, it's rude to blame your customer for using the service you provide as contracted.
This is called double-dipping.
So - Netflix is paying their fair share. You are paying your fair share. If your ISP can't handle the traffic demands of their customers, they need to suck it up and provide the capacity that you're paying for.
Latency is a measure of how long it takes a packet to go from your computer to some other computer and back. People usually measure this by a ping test
Unloaded latency (when there isn't much traffic on your connection through your link to the ISP) generally ranges from 5 msec to 50 msec, depending on what kind of connection you have (fiber, cable, dsl, cellular).
Loaded latency is measured when there is traffic on the link. A good router will minimize the additional latency, but many routers will add a lot - between two and five seconds (2000 to 5000 msec). That's why you lag out in games.
In addition to the new Fast.com test, there are good tools for measuring loaded latency listed on the Bufferbloat page: https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bu...
I just saw the National Weather Service offers new API info. I have no idea whether this meets your need: https://forecast-v3.weather.gov/documentation?redirect=legac...
In either case, neither queue builds up in the home router -- that is exactly the problem, since we can't control the queue size in the modem or ISP! But by traffic shaping to 3.25/0.85 Mbps in the home router (either manually or automatically with fq-codel), we force the queues to build up there, thus giving us control.
YES! I understand what you're saying. (I had been treating the cable/DSL links and their terminating equipment as a black box; your description was being far more specific. Thanks for sticking with me to make yourself clear.)
And as you point out, the fix in all cases (at least, until our ISPs get serious about solving bufferbloat) is to force the home router to take control of the buffering by shaping the traffic a bit below the actual link speed. Thanks.
Yes, but... A "bottleneck" occurs wherever there is a high speed link going to a lower speed link. In your case, the 100 Mbps Ethernets in your home feed through the router to the 1 Mbps upstream your ISP provides. That's another bottleneck.
At this point in a conversation, I always recommend people measure their actual network, to see if they're happy with the situation. If it's good, then everyone's happy.
What results to you get from the DSLReports Speed Test (www.dslreports.com/speedtest)? It measures latency (lag) during the download and upload parts of the test, and will show if your router (or your ISP's router) is buffering too much data, and giving you undesired latency. Best regards.
All routers are doing the same. EU Law.
Citation still needed.
a) I reviewed your eur-lex.europa.eu link, and didn't find anything that prohibits installing other firmware. Please cite the chapter/article that's relevant.
b) re: TP-Link firmware. It is true that TP-Link has added a technical means that makes it impossible to install certain other TP-Link firmware (the "downgrade" they mention). However, there are other firmware distributions (LEDE/OpenWrt/DD-WRT/others) that have the capability to install over the TP-Link factory firmware.
I'm sorry, but I have to disagree. The laws of physics (of the internet) dictate that the piece of equipment at a bottleneck is responsible for handling bloat/congestion on its own end. Your ISP's head end/DSLAM/etc. has an opportunity to prevent download bloat (with data being sent to the home), but it has no way to know what's happening on the consumer end of the link.
Your router at home needs to have the same logic: it needs to control the buffering of data being sent toward the ISP. fq_codel/cake actually measures the time packets dwell in the queues (their "sojourn time") and drops/marks with ECN some of the packets of flows that build up a queue.
A cool feature of fq_codel/cake qdisc in LEDE is that it can control bloat in both directions. It imposes a bottleneck that's slightly (a few percent) below the actual ISP download link speed so that traffic queues up within your home router (instead of at the DSLAM/head end). Consequently, it can do the fq_codel/cake algorithm on the download (as well as the upload) direction, keeping your link unbloated at a very small loss of link speed.
And there's good news on the Wi-Fi front as well. Many of the same people who brought us fq_codel/cake have new algorithms for "Making Wi-Fi Fast".
Their paper tells how they've reduced bufferbloat as well as increasing "airtime fairness" in the Wi-Fi stack. The paper, "Ending the Anomaly: Achieving Low Latency and Airtime Fairness in WiFi", is now available as a preprint at: https://arxiv.org/abs/1703.00064 The conclusion of the paper states:
"We have developed a novel two-part solution to two large performance problems affecting WiFi: Bufferbloat and the 802.11 performance anomaly.
"The solution consists of a new integrated queueing scheme tailored specifically to eliminating bufferbloat in WiFi, which reduces latency under load by an order of magnitude. Leveraging the queueing structure, we have developed a deficit based airtime fairness scheduler that works at the access point with no client modifications, and achieves close to perfect fairness in all the evaluated scenarios, increasing total throughput by up to a factor of 5.
"Our solution reduces implementation complexity and increases accuracy compared to previous work, and has been accepted into the mainline Linux kernel, making it deployable on billions of Linux-based devices."
Fun reading! Oh, and it's available in LEDE today.
Yes, but it can occur on both ends of the bottleneck. Your ISP's equipment has a bottleneck link toward your home, and can have bloated buffers on that side. But if your router doesn't handle the uplink well, it'll be bloated as well.
Fortunately, SQM is available (from LEDE) for many MikroTik routers, so there may be a way out for the GP.
Sure, you could make yourself crazy trying to prioritize all the traffic with varying kinds of flags.
Or you could just let an algorithm (SQM - fq_codel or cake) automatically determine which flows are sending more than their fair share of traffic into the bottleneck link, and offer backpressure to slow those applications down, so other applications work well.
re: net neutrality. That applies to whether my ISP (or any provider upstream in the Internet) should prioritize packets/data from various sources toward (or from) me. The answer should be, No. My contract with my ISP is that I get X mbps of data. The contract doesn't say the ISP will provide higher priority or data rate for service Y over service Z...
SQM has no effect on net neutrality: I'm simply configuring my own router to prioritize the data that I'm requesting (or sending). My ISP should handle them in a totally neutral way...
Not so - Bufferbloat can occur everywhere there's a bottleneck link.
Your home router's link to the ISP is likely to be one, and most don't have any mitigation, and suffer from high latency. (And various ISP's equipment has bufferbloat for data coming toward your home.)
Some sites suggest that you knock your bandwidth down to 70% of the link speed to "leave some headroom" for other packets. That's fine if you want to give away 30% of your capacity.
SQM (implemented through fq_codel or cake qdisc's) takes off a few percentage points from the rated link speed, and achieves great results with minimal setup and configuration. Check out https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a... for more info.
Yes, Don't Congest. But you can spend your life marking various kinds of traffic (many apps already do mark it, as someone down-thread mentioned), and pray that your router (and remote equipment) handles the markings properly.
Or just use SQM with Cake qdisc. It actually looks at the "sojourn time" for packets in various streams/flows of data and offers backpressure for sessions that are generating queues of data: all the other flows "go right through" without a lot of other settings.
I'm surprised that no one has mentioned bufferbloat yet. In most home routers, people can see seconds of delay when there's significant other traffic on the link. (And remember, "other traffic" can be ordinary web browsing, where pages average 2 mbytes these days...)
The latency (and jitter) caused by bufferbloat has been a solved problem for almost five years: fq_codel, and the newer cake qdisc's are are in the Linux kernel, and people's home routers now could install LEDE/OpenWrt (https://lede-project.org). Two links:
What can I do about bufferbloat? https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a...
Test your network for bufferbloat: http://dslreports.com/speedtest
@pktgen Thanks for the good note. You've nailed the science behind this and the proper fix. In your note below, you also note that Ubiquiti router firmware has fq_codel/cake.
I'd like to mention that both LEDE (www.lede-project.org) and OpenWrt (www.openwrt.org) were the platforms used for developing and testing fq_codel/cake. That means that people may be able upgrade their existing router to eliminate bufferbloat.
My advice: if you're seeing bufferbloat (and a great test is at www.dslreports.com/speedtest) then configuring fq_codel or cake in your router is the first step for all lag/latency problems.
I use Password Wallet (http://www.selznick.com/). It's not sexy, but has decent web browser integration, and it has never failed for me. (More specifically, it does a couple things well, not trying to do everything for everyone.)
Yes, the BBR developments seem to help TCP connections a lot.
There is also a lot of good research going on in the "Make Wi-Fi Fast" project. See https://www.bufferbloat.net/projects/make-wifi-fast/wiki/
For concrete results, see June 2016 paper: "Fixing the WiFi Performance Anomaly" at https://blog.tohojo.dk/2016/06/fixing-the-wifi-performance-a...
Read the archives of the "Make-wifi-fast" mailing list at: https://lists.bufferbloat.net/listinfo/make-wifi-fast
If you want to use bleeding edge LEDE code (spinoff of OpenWrt) to see if your Wi-Fi is faster than mine, try: https://kau.toke.dk/lede/airtime-fairness-builds/ar71xx/gene...
Get your lag down to milliseconds no matter how you load your network.
OpenWrt 15.05.1 ("Chaos Calmer") is stable, and quite secure as shipped. Using the default packages, or a few well-known packages as add-ons will be fine from a stability, security, and UI standpoint.
OpenWrt Trunk ("Designated Driver") development doesn't seem to be going as quickly as it was before LEDE split off. It's still a development trunk version, and not guaranteed to be stable.
LEDE is just getting off the ground. Its nightly builds change (improve) rapidly, but you may not want your family (spouse, kids) to rely on it right now. It will have the same (or better) stability, security & UI characteristics as 15.05.1 when they ship their first stable version. There's no schedule yet, but look for an announcement in the new year.
For your stated requirements, I would recommend (and am personally running) OpenWrt 15.05.1
strapdown.js is pretty cool... Just create a static file with your Markdown text within <xmp> tags... http://strapdownjs.com/
Oh, yes. And no "contact us" link...
Regrettably, these folks use the "cream of the crop" technologies, but get the wrong answer. (7mbps/768kbps DSL shows up as ~2mbps/560kbps...)