dtaht

The difference between A and D in the 68000 was sometimes bothersome. I think riscv is cleaner to think about.

That said, I loved the zero page on the 6502...

Too bitter. I referenced a little of that "adventure" here, in 2021... gfiber was attempting to restart with refreshing their now obsolete hardware... https://blog.cerowrt.org/post/trouble_in_paradise/

They laid us all off. They had huge plans - millions of users! Then they intersected reality in KC where all people wanted was 5Mbit service and free TV... There were many, many people working to perfect the settop box for example. We got fq_codel running on the wifi, we never got anywhere on the shaper, the plan was to move 1+m units of that (horrible integrated chip the comcerto C2000 - it didn´t have coherent cache in some cases), I think they barely cracked 100k before pulling the plug on it all....

and still that box was better than what most fiber folk have delivered to date.

At least some good science was done about how ISPs really work... and published.

https://netdevconf.org/1.1/talk-measuring-wifi-performance-a...

I´m pretty fond of ¨Ending the anomaly¨: https://www.usenix.org/system/files/conference/atc17/atc17-h...

The users that call less post-libreqos are the gamers & zoomers.

I am sorry you are are so down on the lack of motivations care for customer service that many ISPs have. I am thrilled by how much libreqos's customer base cares.

I agree that the phrasing in the article is a bit confusing there!

Pie had a severe problem in the rate estimator which was fixed in 2018, in Linux, at least:

https://www.sciencedirect.com/science/article/abs/pii/S13891...

Pie's principal advantage is that it is slightly easier to implement in hardware, it's disadvantages are that it does tail drop, rather than head drop, and struggles to be stable at a target of 16ms, where codel can go down to us and targets 5ms by default. I haven't really revisited pie since the above paper was published.

COBALT in cake is a codel derivative. It is slightly tighter in some respects (hitting slow start sooner), and looser in others (it never drops the last packet in a queue, which fq_codel does.fq_codel scales to hundreds of instances and 10s of thousands of queues, still aiming for 5ms across that target, where it would be easier to essentially DOS that many instances of cake with tons of flows.

In a lot of cases older is better.

See also https://github.com/lynxthecat/cake-autorate for an active measurement tool...

I would replace the home router with an OpenWrt router.

I share your pain. I really really really share your pain.

BITAG published this a while back.

https://www.bitag.org/latency-explained.php

It's worth a read.

big fan of flent.org and this tool, written in rust - is coming along smartly.

https://github.com/Zoxc/crusader

Being one of the authors of fq_codel, cake and to a small extent, LibreQos, I remain boggled after 15 years of trying, to get these simple points across..

Have some laughs: https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-...

OpenWrt depreciated pfifo_fast in favor of fq_codel in 2012, and have not looked back. It (and BQL) is ever present on all their Ethernet hardware and most of their wifi, no configuration required. It's just there.

That said many OpenWrt chips have offloads that bypass that, and while speedier and low power, tend to be over buffered.

We have seen small ISPs get LibreQos running in under an hour, which includes installing ubuntu. Configuring it right and getting it fully integrated with the customer management system takes longer.

We're pretty sure most of those ISPs see reduced opex from support calls.

Capex until the appearance of fq_codel (Preseem, Bequant) or cake (LibreQos, Paraqum) middleboxes was essentially infinite. Now it's pennies per subscriber and many just a get a suitable box off of ebay.

I agree btw, that how to monitor and scale is a learned thing. For example many naive operators look at "drops" as reported by CAKE as a bad thing, when it is actually needed for good congestion control.

speedtest.net added support for tracking latency under load a few years ago. they show ping during up/dl now. That's the number to show your colleague.

However they tend to use something like the 75th percentile and throw out real data. The waveform bufferbloat test does 95% and supplies whisker charts. cloudflare also.

No web test tests up + down at the same time, which is the worst scenario. crusader and flent.org's rrul test do.

Rathan than argue with your colleague, why not just slap an OpenWrt box as a transparent bridge inline and configure CAKE SQM?

The rate limiter in question: https://github.com/cilium/cilium/issues/29083

It really is astounding to me how so many still do not understand that tcp is not a function call, or behaviors like slow start and congestion avoidance.

Recently a new rate limiter for TCP went by that was so terribly, terribly broken, and I cannot help but imagine that most of the containers of the world suffer from Bufferbloat in general.

My favorite book of his was actually one of his earliest - Zodiac.

Thank you. I enjoyed the sense of play and this story more than anything I have read about the internet in a long time.

I am def burned out, and need to come up with something frivolous. I am reminded of Richard Feynman´s story of spending 10 years depressed after the war, and him finding joy of the physics in a spinning disk one day at lunch, so he could disregard what he had done before.

You are citing amdahl's law: https://en.wikipedia.org/wiki/Amdahl%27s_law

one of my sadnesses is so few tests, except those designed by the bufferbloat project, test up + down + latency at the same time.

I like this new one: https://github.com/Zoxc/crusader/releases/tag/v0.0.10-testin.... My favorite remains the rrul test out of the flent.org suite.

Most web speed tests have nearly no predictive power for testing videoconferencing quality, although I am delighted speed test.net cloudflare and others have added a "responsiveness" metric that too few see... https://blog.cerowrt.org/post/speedtests/

most games run at 64Kbits/sec or 120Kbits and FQ helps those "mice" flows a lot, by giving them their own lane.

If also, the greedy protocols like TCP or QUIC back off on filling up buffers (via an AQM) before a gamer would notice (20ms of jitter/buffering is about right, 60ms is teetering on wrong) gamers and downloaders can co-exist happily on the same link.

lastly, it is not just TCP that AQMs can control, but any TCP-friendly congestion control - Quic - which is udp based - plays by those rules. Most of our videoconferencing applications in their bandwidth seeking phase, also.

QUIC's original congestion controller was based on Reno. Other versions use BBR or Cubic congestion controls:

https://datatracker.ietf.org/doc/html/rfc9002

Thank you for the steer to that article, it has some errors.

ECN is not in use at starlink (yet), so far as I can tell. The AQM in place just uses conventional - and timely - packet drop, which is the principal signal for congestion control across the internet. (so I would say " alternatively" in the Wikipedia article about ECN, and drop principally)

To this day, so many people (including the early engineers at starlink), do not get that dropping packets is how we manage latency across it for greedy flows. I have given so many talks about how this works - lose 2 minutes of your life to this, please: https://www.youtube.com/watch?v=TWViGcBlnm0&t=995s

Starlink folk now deeply get that buffering for 100s of Ms is counter productive, and have put in play many "bufferbloat" mitigations over the past year, focusing on P99 metrics in particular.

...

More importantly than AQM, their stuff does "flow isolation" and "Fair queuing" so that fat flows interfere with lighter weight flows less. https://en.wikipedia.org/wiki/Fair_queuing

The starlink wifi now does fq_codel in particular, which combines fair (flow) queuing with the codel AQM. While I do not have any benchmarks of their implementation it should show up on tests where two or more stations are trying to "do stuff" at the same time.

...

There has been a lot of hype about ECN lately (because it does not require dropping packets) but it needs to be enabled at the endpoints, and at the congested routers in-between running an AQM algorithm that also has it turned on. I do keep hoping more enable it on their OSes (apple has it on for some sites, Apple Quic does now), and more routers enable it also.

Stunningly widely supported is RFC3168 ECN across 68% or so of all websites, but few clients turn it on. Here's how you can try it:

https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN...

If you have a supported router leveraging SQM ( on any tech, fiber, docsis, what have you) : https://openwrt.org/docs/guide-user/network/traffic-shaping/...

I note that ECN has some major drawbacks vs a vs drop - a bad guy can abuse it, in particular, which is in part why it is not on by default. If

WRED is so obsolete...

https://en.wikipedia.org/wiki/CoDel

It is fq_codel on the wifi: https://en.wikipedia.org/wiki/CoDel

I had hoped this talk would stir more discussion than it has so far.

I don´t know. That 45% figure for quic was news to me.

Don´t give up. Ask for RFC7567 everywhere.

+10!

Mosh, over wifi, is a sanity saver.