HN meta: 99% packet loss on news.ycombinator.com

https://news.ycombinator.com/item?id=44972745

by eqvinox • 11 months ago

4 5 11 months ago

news.ycombinator.com

I'm currently seeing >99% packet loss to news.ycombinator.com (2606:7100:1:67::26):

  13. br06-te1-2.lwdc.americanis.net                   0.0%   231   14.3  17.2  14.0 172.5  19.5
  14. sdtc.sd-me01.te0-1.host1.20546.americanis.net   99.6%   231   12.9  12.9  12.9  12.9   0.0
  15. 2606:7100:1:67::26                              99.1%   231   11.8  11.8  11.8  11.8   0.0

Checked from both US and EU connections, same result, would be a very odd coincidence if it was a problem on my end. Trace looks like the uplink provider is having issues?

IPv4 seems to be unaffected.

Bender3• 11 months ago

Packet loss over ICMP can be artificial based on the backplane load of routers, ACL's and artificial rate limits. They will de-prioritize responding to ICMP and fail to respond if CPU load is high but that does not mean the packets are not being forwarded 100%. Linux and BSD also have knobs for this behavior as well. HN is BSD.

Are you seeing TCP retransmits to port 443?

    for i in $(seq 69);do nc -vz -w1 news.ycombinator.com 443;sleep 2;done

Do that at the same time as watching

    watch -d -n4 --no-title 'nstat -a|grep -Ei "ret|los"'

It would be most useful to do this from your location and also from VM's on a few different providers in different locations to find which thing is not like the other. The 'nc' I am using is part of the nmap distribution. To force IPv6 replace the name with the IPv6 address.

eqvinox2• 11 months ago

Packet loss over ICMP can be artificial based on the backplane load of routers.

This doesn't generally apply to the end host; note the last hop in the trace I posted is the web server itself. Also, the second last hop having very similar loss% makes it likely that neither of the two systems is hitting rate limits (as they would be different).

Are you seeing TCP retransmits to port 443?

If the webpage loaded normally, I wouldn't have investigated to begin with ;).

Right now, the loss on TCP SYNs is about 70%. (mtr -T -P 443) Second to last hop still has roughly the same loss. Something's h0rked at americanis.net.

Bender3• 11 months ago

If you are confident where the loss is then the only thing you could really do within HN is email Daniel (dang) and ask him to replicate the issue and open a ticket with their upstream provider to check routes and for losses. hn@ycombinator.com

Outside of HN I suppose you could check the Nanog mailing lists [1] and chat servers to see if there is any discussion ongoing for those networks.

[1] https://lists.nanog.org/mailman3/

HN meta: 99% packet loss on news.ycombinator.com

Related Stories