Ask HN: How can I tell if a site is being rate limited?

https://news.ycombinator.com/item?id=23017236

by caseyslaught • 6 years ago

48 26 6 years ago

news.ycombinator.com

I live in a country with less than ideal Internet freedom standards and I have a suspicion that a popular publishing site is being rate limited. How can I know for sure? Is it possible to show a smoking gun?

chefkoch• 6 years ago

You could test from vps in another country?

Raed667• 6 years ago

My country used to heavily censor internet pages, do finishing attacks on Facebook, Gmail etc..

The easiest way to prove it is by using a VPN or a proxy. Check with different ISPs, also try mobile vs home.

caseyslaught4• 6 years ago

I can definitely notice an improvement in performance when I use a VPN, however it would be cool to see where things are being delayed. Is traceroute or curl something that could help here?

Retric• 6 years ago

Yea, if it’s faster via VPN that’s a strong hint. If you want proof you need to examine the internet between you and the site. If a VPN connection that uses a specific router to to the site is fast, your connection through that same router is slow, and you can connect to some other site through that same router and get a fast connection, then that’s a smoking gun that your connection is deliberately slowed.

That said, it’s not easy to prove it’s your country/ISP doing the slow down vs that website slowing down connections to your country.

rdbell• 6 years ago

My company has a residential proxy network that lets you test endpoints with requests from residential broadband ISPs all over the world. Some of our customers are using our network for exactly this use case: https://packetstream.io

sairamkunala• 6 years ago

its possible that your ISP has a bad peering connection. Checking with your ISP usually helps understand what is going on.

It could be the bandwidth limitation of your ISP itself.

Sites like Github are slow when accessed from few countries. It's just that their servers are located mainly in the US(afaik).

caseyslaught4• 6 years ago

I ran a curl test and noticed that the TCP connect time is where things are getting hung up, taking over 80 seconds! Does anyone know if this is indicative of deliberate rate limiting or just a bad peering connection?

toast0• 6 years ago

If it takes 80 seconds to TCP connect, something is pretty wrong.

tcpdump / wireshark would likely show lots of retransmits. Traceroute/mtr may give you an idea of where the loss starts. If you can find servers on similar hosting, you can get some idea of if it's targetted or not.

bbulkow• 6 years ago

Possibly neither, that is, there are many possible reasons.

for example, stateful firewalls can only track a certain number of connections. If you don't have a public IP, or if they are trying to protect you, it may be necessary to wait for a connection resource to come available. Connections are generally expensive, different from data transfer, for these and other reasons, such as port starvation, syn flood defenses inappropriately targeting you, so many others.

mtlynch• 6 years ago

I don't have personal experience using it, but I believe Ooni Probe is designed for your exact use-case:

https://ooni.org/about/

caseyslaught4• 6 years ago

Thanks for the recommendation!

leowoo91• 6 years ago

If implemented with respect to the http code, it should say 429 in the developer console.

sfkdjf9j3j• 6 years ago

State internet restrictions aren't implemented in the application layer.

bmm6o• 6 years ago

That's not the kind of rate limiting he's asking about. He is talking about the data stream being throttled.

crazygringo2• 6 years ago

Just curious -- if a country didn't want a site to be seen, why would they rate-limit it instead of just blocking connections completely?

Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?

Liru• 6 years ago

People's attention spans are ridiculously short. If the person is actively looking for the content, sure, that obviously won't work. However, if it's a more casual browser, then going to a site that takes a long time to load between different pages is definitely a way to get them to not visit it any more. This even works on tech-savvy people, as opposed to just blocking a site which will just get them to use a proxy.

Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?

China and Russia do this with a lot of western sites, from what I've heard.

jedberg• 6 years ago

It would be too obvious. If you can't get to the site you know it's blocked. If it's just super slow all the time, you won't know why and may just get frustrated and leave.

opportune• 6 years ago

Yes: https://news.ycombinator.com/item?id=22541960

I have seen this actually happen IRL

crazygringo2• 6 years ago

Thanks so much for the link. That was really educational, answered my question completely.

7ewis• 6 years ago

Would mtr[0] help diagnose this?

[0] https://github.com/traviscross/mtr

willcipriano• 6 years ago

One possible method: Write a script to request the site continuously and see if it fails to respond with any regularity. From there you can start dialing it in and determine what the rate limit actually is. A rate limit should be consistently reproducible unless it's implemented in a clever way.

W4phle_Stomp• 6 years ago

Have some fun at the same time: 1) get a vpn account (proton, other) 2) hack your worst enemy's wifi with your favourite tools 3) while outside your enemy's home, boot a linux live distro 4) change your computer's name to your enemy's name 5) connect to the enemy's wifi 6) try to visit all the sites banned by your country 7) insult your country's leader in a public forum using your enemy's name 8) test the speed of the suspected rate-limited site 9) set-up your vpn account settings 10) using your vpn, re-test the rate-limited site 11) subtracting a fair percentage slow down because of the vpn, gauge if there's a significant difference between the two tests 12) turn off computer, go home, enjoy tasty beverage 13) grab some popcorn, go watch your enemy's house from a reasonable distance 14) post video on youtube of your enemy getting busted, using your favourite heading

If that's all too much fun for you, then skip most of it and only do #s 8 to 11 from your own network or favourite hot-spot.

DeathArrow• 6 years ago

He should also modify the said enemy's browser cache for this to work well.

caseyslaught4• 6 years ago

This is very insightful. Tip of the hat to you!

meesterdude• 6 years ago

The thing about rate limits is, there are often hard limits. Try querying the site a bunch and see what the data transfer rate is. if it's consistently incapable of going above say, 128K/sec, you've got your rate limit.

Other times, they'll allow for bursts but sustained get rate limited - these can be detected just the same depending on their approach; if it's per connection (and not IP) you may have to find a big file to download.

It also helps to establish that traffic to other sites is not rate limited, and to use 3rd party checkers that check for connection rate across the globe. Or DIY and spin up a bunch of EC2 machines that are geographically diverse.

rozab• 6 years ago

I don't know about this, I've heard that limiting is often intentionally inconsistent because this makes people less likely to visit. I think Harlow did some similar stuff with his monkeys.

Ask HN: How can I tell if a site is being rate limited?

Related Stories