Cloudflare misidentifies Hetzner IPs as being located in Iran

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8121#note_1237201726

by doruk101 • 2 years ago

204 219 2 years ago

Yeah, google does it too. I could not use certain Hetzner IPs to download container image on my kubernetes nodes at all. Even the official registry.k8s.io registry is hosted on Google Cloud Services and basic stuff like the pause image cant be pulled.

kodama-lens• 2 years ago

I can confirm this. All Google container registries, including the official k8s repos are unaccessible via some hetzner ipv4 domains.

There is a GitHub issue that also covers the problem and it states you should report thos IPS to their support. I did but support says they can't do anything until the ip region list is updated.

IPv6 as a workaround is also difficult because some of the image I need are on GitHub and they are still not ipv6 accessible

wiredfool• 2 years ago

For a while, Google was blocking IPv6 from Linode, to similar effect.

preisschild2• 2 years ago

I noticed some IPv6 addresses were getting blocked too.

We reported a lot IPs to Hetzner, but since we use autoscaling new blocked ones just kept on appearing.

Jach2• 2 years ago

Google's IP to location mapping is so bad it has to be intentional. I was in Japan and using my home network as a VPN quite a bit, after a while Google decided my home comcast IP had to be located in Japan. Even though others in the household were still there, they started getting default-Japanese pages on google/maps/youtube/... It didn't fix itself back until a couple weeks after I got home, even filled out https://support.google.com/websearch/contact/ip

ipaddr• 2 years ago

They finger print your browser. You need to vpn to your home and serve from your US browser not tunnel traffic back to your Japan machine.

38• 2 years ago

I mean, so? Why should it matter where they are located?

tyree731• 2 years ago

You may be surprised to discover that services will filter traffic by location.

themoonisachees• 2 years ago

In my previous jobs we didn't have any business in china and banning all IP ranges was a cheap an easy strategy to remove 50% of unsuccessful login attempts.

reisse2• 2 years ago

A lot of US resources ban traffic outside the US. Or, at least, from "suspicious" or "sanctioned" locations. Some ban EU due to GDPR.

You never know such things when you are in US though...

dathinab2• 2 years ago

but the traffic is _clearly coming from Germany_, the issue is that cloudflare/google have tagged certain ip addresses as Iranian no matter where the traffic actually originates from

VWWHFSfQ2• 2 years ago

My servers ban huge swaths of IPs from certain places that originates enormous amounts of spam, scanners, and other nefarious traffic. It's very effective

FredPret3• 2 years ago

Sturgeon’s law [0] apply to all sorts of things, including web visitors

[0] https://en.m.wikipedia.org/wiki/Sturgeon%27s_law

Fokamul• 2 years ago

So you ban US and China, aka two places where most spam, ddos and malware is coming from. Right?

ajsnigrutin2• 2 years ago

So, all the cloud, vps, and hosting providers?

kiwijamo2• 2 years ago

If I followed your strategy I would be blocking all of Google. Back in the days I operated my own mail server >50% of all spam was from Google USA... YMMV.

Dalewyn• 2 years ago

US sanctions prohibit transfer of goods, technologies, information, etc. to Iran.

As a company, this means BSTS (better safe than sorry) CYA (cover your ass) measures for good or worse.

Narhem• 2 years ago

Curiosity may get me on this one, but is sharing information (such as this post/comment) an example of transfer of information (to potentially all countries)?

greyface-• 2 years ago

It's the interpretation of some cloud providers that exchanging datagrams with entities in OFAC-sanctioned countries constitutes a prohibited transaction.

londons_explore4• 2 years ago

Which is plainly stupid.

They should interpret the law to mean "We will treat every request from Iran as a non-paying customer, and won't offer anything outside the free-tier"

Even if that isn't the way was written, it is plain that it falls within the intent of the law, and is beneficial to US businesses.

Animats• 2 years ago

The actual sanctions are complicated.[1]

There's a big list of allowed Internet activity between the US and Iran.[2] It is explicitly US policy to not cut off Iran from the Internet. The State Department wants people in Iran to get info from the outside world. However, the US does not allow US domain registrations or web hosting "for or on behalf of the Government of Iran".

The Office of Foreign Assets Control can be queried for case by case info. That's appropriate here.

[1] https://www.ecfr.gov/current/title-31/subtitle-B/chapter-V/p...

[2] https://www.ecfr.gov/current/title-31/subtitle-B/chapter-V/p...

greyskull• 2 years ago

Might be pertinent to suffix this with (2023), though I see there are still recent replies

jkaplowitz• 2 years ago

It's a still-unresolved issue as far as I know; the linked ticket was only closed last year because Gitlab has no control over it as long as they want to continue using Cloudflare. The companies which do have control over it have not fixed it so far.

tgma• 2 years ago

My theory is lots of people who want to circumvent Iranian internet censorship rely on tunnels/VPNs hosted on Hetzner, which correlates those IPs with `Accept-Language: fa` and GPS locations collected from Android or other similar behavior.

Alex-Programs• 2 years ago

Yeah, I had the same theory when Google did this with a free-tier VPN IP that was in Turkey. It claimed I was in Tehran - and, when I looked at the map of servers, the Turkish server I was connected to was the closest to Tehran.

lutoma• 2 years ago

I think a more likely explanation is that Hetzner just acquired some IPv4 address ranges that were previously used in Iran

rany_2• 2 years ago

I think that might be less likely given the trade restrictions. There's no way an Iranian ISP just gave Hetzner those IPv4 addresses free of charge.

rany_2• 2 years ago

That's almost certainly the case. I use Tor semi-regularly and many Tor exit relay IPs are identified as being in Iran which is just not possible.

scandox• 2 years ago

Would be an easy way to conduct an adhoc trade war...AWS doesn't need competition from a pesky German host let's just make things faintly awkward...

osiemens• 2 years ago

I wonder if this is related to something I found when I moved my hosting from DO to Hetzner: https://on-no.net/posts/moving-providers-and-tainted-ips/

TL;DR is that the IP that my new instance was assigned had previously been used as part of an advertising CDN based in Iran. It wouldn't surprise me if this is some game of whack-a-mole between interested parties who are at turns applying and attempting to evade blocks.

paulv• 2 years ago

Does this kind of thing affect Hetzner IPs in their US datacenters?

jsheard• 2 years ago

It looks like they use different ASNs for the US datacenters, so probably not in this case. Nuremberg, Falkenstein and Helsinki all share the problematic AS24940 block mentioned in the OP, but Ashburn is on AS213230 and Hillsboro is on AS212317.

dathinab2• 2 years ago

it's pretty absurd that cloudflare can just effectively cripple a cloud provider by tagging part of their IPv4 range as Iranian and not fixing their issues in over a year (and AFIK have no intention to fix them at all)

like I wonder if Hetzner has any way to legally force them to stop misclassifying their IP

dools• 2 years ago

Kinda seems like it might have legs as a defamation lawsuit...

amatecha• 2 years ago

What's absurd to me is that Cloudflare gains more and more control over the internet, by people voluntarily submitting to its domination.

My favorite is trying to go someone's random blog with like 5 posts (because they have a singular post about the technical topic I'm trying to figure something out about) and I can't access the site because Cloudflare has decided my locked-down Firefox ("resist fingerprinting" + strict privacy mode etc.) running on OpenBSD is somehow malicious. So much for the open web. (nevermind the audacity that "we can't spy on you sufficiently" is enough to serve a 403 Forbidden response header)

literallycancer4• 2 years ago

Might have something to do with how that particular website is using Cloudflare.

anonzzzies10• 2 years ago

It is extremely hard to stop DDOS attacks without CF; my hoster has DDOS protection, but when there was a very large attack on our site, only CF could remedy it, and did so immediately when we panicked-moved dns and switched on bot fight. Entire attack that my hoster couldn't stop was gone. How do you do this without CF if you are a small company?

TheTr1ckt3r• 2 years ago

This whole issue of blocking Iranian IPs and not allowing them to download Docker containers for ‘legal’ reasons is ridiculous. Additionally, trying to detect and ban VPNs used by Iranians, which will affect the next user of that IP, is equally absurd

appendix-rock• 2 years ago

What do you suggest, then? What’s your legal opinion?

citrin_ru• 2 years ago

1. avoid geoip blocks because geoip is inaccurate 2. When maintaining geoip don’t mark servers physically located in DE but used by a foreign company as located somewhere else because it will quickly go stale and misleading in the first place

RadiozRadioz• 2 years ago

I'm frequently reminded how thankful I am to live in a country with a strong, positive international reputation. Even ignoring actual quality-of-life stuff associated with where I live - simply not being from a country with a "dodgy" reputation makes many things so much easier.

I don't have to think about blocked websites. Companies accept my payments. Couriers ship to me. With my passport, I walk straight to the front of the fast lane, past the large queue of people who didn't happen to be born somewhere rich, western and politically stable.

I don't take it for granted, and it makes me sad that this distinction exists.

Onavo• 2 years ago

It's more that you are on the right side in a unipolar world. When the world shifts to multipolarity in the next few years, the problem will solve itself.

kiba• 2 years ago

Multipolarity is a more dangerous world, as we have seen Russia asserts itself at the expense of Ukraine.

01HNNWZ0MV43FF• 2 years ago

I'm a globalist and all but when people say "multipolar" doesn't that usually mean "the USA shouldn't rule everyone, I want to also rule over some countries "

rtsil• 2 years ago

I doubt the "next few years", and if the world shifts to multipolar, it won't solve the problem, it will just move everyone to the "bad side" where frictions big and small abound.

nradov4• 2 years ago

Nah. The other "pole" looks to be China. They aren't going to treat random people from poor countries any better.

ajsnigrutin2• 2 years ago

I live in a small EU country.

There are many, many american sites that just block the whole EU IP ranges becaus they don't want to deal with GDPR.

ttt3ts• 2 years ago

I have done that exact configuration for several of my clients who didn't realize any/much revenue in the EU. For them it was the obvious best move but I wish there was a better option.

poincaredisk• 2 years ago

Are there? I only someones stumble upon some medical website that redirects me to a "tracking free" empty static page[1] if I come from Europe and opt-out of cookies (which I always do anyway). Maybe we visit other parts of the internet, I don't read a lot of non-IT English things.

[1]A big troll that I respect.

throwaway920102• 2 years ago

they don't want to deal with GDPR

or cannot afford to. add in DSA and DMA as additional burdens.

menacingly2• 2 years ago

I'm surprised I don't see it more. You can't impose a regulatory burden more troublesome than your traffic is worth

_DeadFred_• 2 years ago

Hey, this is me, I do this.

skissane3• 2 years ago

I’ve been noticing more and more US state and local government websites blocking traffic from outside the US. (And I’m not talking about traffic from North Korea, I’m talking about traffic from ANZUS/AUKUS/FVEY ally Australia.)

It seems stupid because just because someone is overseas doesn’t mean they can’t have valid business with a US state or local government. Maybe they are an American who is travelling and has to attend to some official business back home while they are away. Foreigners are allowed to purchase US real estate and incorporate companies in the US, which gives them heaps of legitimate reasons for interacting with local and state governments. In part due to these kinds of issues, many use some local agent in the US to handle government interactions for them, but a person can have valid reasons to engage directly.

grishka10• 2 years ago

I live in Russia and I've never experienced most of the things you're describing. And it's become so much worse after 24/02/2022. We even had Spotify for a year! It was starting to genuinely feel like a first-world country.

Now you have to open a bank account in a different country for foreign companies to consider taking your money at all. The internet is utterly broken. The government blocks quite a lot, AND some foreign services block Russian IPs from their side. I even made a thread about running into Cloudflare's "you're blocked" pages randomly throughout the web: https://mastodon.social/@grishka/111934602844613193

aguaviva7• 2 years ago

Do you think the sanctions are having a significant effect in terms of slowing down the war effort?

paraknight2• 2 years ago

For half my life I had an Egyptian passport, and for the other a German passport. Having experienced both sides, that bit of paper is without a doubt the most valuable thing I own.

It's hard to quantify the kinds of doors it has opened for me. I was able to get a scholarship to study in the UK that covered home/EU rates (a third of international rates, while I might not have been able to get even a student loan otherwise), get government funding for a PhD that would not have been accessible to me otherwise and other grants, travel to international conferences without thinking twice about visas (unlike many colleagues) meeting people that would impact my career and skipping all sorts of and barriers along the way, and never had to worry about deportation because of the EU settlement scheme, easily become a founder (no visa sponsorship needed), and so much more! Even travelling/business in the the middle East, being German rather than Egyptian is an entirely different life, one that my cousins cannot even begin to imagine.

There's a parallel universe where I'm stuck making ends meet in Cairo where I was born, dreaming of a brighter future, feeling all my potential fade away. I know because my immediate family is that version of me - no less talented or worthy of the opportunities I got because of my nationality!

I see the kind of freedom that I have because of that passport as one of the biggest modern injustices.

chipdart10• 2 years ago

I see the kind of freedom that I have because of that passport as one of the biggest modern injustices.

I think you're confusing a vague and abstract problem of "injustice" with a very concrete and real difference in ways different countries manage their public services and institutions.

You only listed personal benefits that a country like Germany provides to their citizens and the higher education institutions built up by the UK, and how it contrasts with the ones provided by Egypt.

Quite bluntly, this is a discussion over privileges. Not injustice, but privileges. I assure you that countless people from Germany, UK, the EU, or anywhere in the world, would desperately want to have access to the same opportunities. Depicting this as a matter of being granted a passport is at best survivorship bias, and at worse an affront to those who had it but still weren't lucky enough to benefit from the same opportunities.

alberth• 2 years ago

This probably wasn’t cloudflare doing per se. It was probably Maxmind, which is the most widely used IP to Geolocation service out that.

And cloudflare uses it as well.

https://developers.cloudflare.com/network/ip-geolocation/

ewpratten• 2 years ago

Hey OP. On behalf of Cloudflare, we take information accuracy very seriously.

I raised the linked issue internally with the team, and they have reason to suspect this has already been addressed.

That being said, if you (or anyone else here) are still seeing this issue occur, please raise a ticket with our support team (https://developers.cloudflare.com/support/contacting-cloudfl...) so we can investigate further.

Thanks :)

ggm• 2 years ago

https://geolocatemuch.com/ is the way.

lxgr7• 2 years ago

Falsehoods lawyers believe about the Internet: You can identify a person (and their jurisdiction) from “their” IP address.

nradov4• 2 years ago

It's not a falsehood though. IP address is a reasonably reliable means of geolocation. Lawyers tend to be more comfortable with gray areas than engineers. Intent counts for a lot in assessing legal compliance.

lxgr7• 2 years ago

From a "best practice"/CYA perspective, sure.

But I'm not a lawyer, and looking purely at the outcome of IP blocks (which is usually that regular people are inconvenienced, but the people such policies are actually designed to keep out just shrug and use a $5/month VPN), I can still say that it looks a bit silly.

gsich• 2 years ago

No need for anticipatory obedience though. If whois says it's not Iran - who cares.

saurik• 2 years ago

But it isn't a grey area: it simply doesn't work. It doesn't matter if it correctly identifies most people: it has to correctly identity most terrorists, and it simply doesn't do that, because if you are a terrorist you just keep rotating through IP addresses on cloud providers and VPNs until the entire service is burnt. It isn't that it sometimes doesn't work: it's that it doesn't work at all when it actually needs to work. We could argue that the services shouldn't let you do that in the first place, but the reality is that services currently do work like that, no one is trying to change that, and if they did try to change it we would all be even less happy with the resulting even-more-powerful surveillance state.

miki1232113• 2 years ago

Falsehoods programmers believe about law: the fact that an identification method isn't 100% accurate means that it has no value

lxgr7• 2 years ago

Maybe this is more of a Europe vs. US observation than a programmer vs. lawyer observation, but I have indeed made the observation that US companies are often satisfied with "identity verification" that would absolutely not fly elsewhere. A PDF of a utility bill as "proof of residency", knowing somebody's SSN as "identity verification"...

Yes, they might be definitionally best practice and accordingly enough from a legal perspective, but I don't see them having any value in actually keeping out bad actors. A fence that surrounds 99% of your pasture indeed has no value if the wolves know where the 1% gap is.

IncreasePosts2• 2 years ago

Ah. So I can log IP addresses that connect to my service and store them forever?

lxgr7• 2 years ago

IP addresses are great for identifying traffic patterns, figuring out where your audience is roughly located etc. as long as you don't use them to selectively block users – since then nobody has a real incentive to "cloak" theirs.

Once you start doing that, you've completely destroyed the measurement, and at the same time you're still not keeping out unintended users – because these will just use a VPN.

To go with an analogy: Imagine a bank enforcing embargo/sanction policies by just asking everyone at the entrance for their name but not checking their ID! You'd get a lot of personal data (since most people won't lie), yet you won't keep any sanction evaders out.

ksajadi• 2 years ago

This is not limited to Cloudflare. Google has the same issue and it turns out the IPs were being used by the Iranian hosting companies connected to internet surveillance but they keep moving around. So far we only observed this in Hetzner German DCs, which is consistent with the news about illicit activities by Iranian companies in Germany, two years ago during the last uprising against the Iranian government (the Woman, Life, Freedom movement)

We also wrote about this https://blog.cloud66.com/hetzner-connectivity-issues-due-to-...

thundergolfer• 2 years ago

Happened to use with GCP too. We had Oracle Cloud instances being flagged as from Iran and had to file forms with them to get them to not block the IPs.

TZubiri• 2 years ago

elwebmaster2• 2 years ago

It should be illegal for providers to override the location information provided by the owner of the IP. Hopefully the FTC will look into this abuse. In the real world this would be the equivalent of me putting my shipping address on an order but the store deciding to ship it to some random place because they “believe” that’s my actual address.

cr125rider2• 2 years ago

No. Don’t trust client data. Caller ID is garbage because they do this.

reincoder• 2 years ago

I will describe what we do at IPinfo to avoid such a messup. First of all because we do active measurements and our data is usually less prone to errors like this and when it comes to IP location it is as good as it gets.

We have a support team active 247. Then is the issue of update rollout, when things goes wrong (rarely if ever) we can push data updates immediately. We work with our customers and users and try to push immediate fixes.

But the most important thing in my opinion we do is this comment itself. If things go wrong we will address it before you come to our support team.

betaby2• 2 years ago

Is Apple store working in Iran? For example, Apple store is working in Russia.

I genuinely do not understand how logic works between 1.sanctions 2... 3.let's ban some IPs. What is the chain of reasoning happens on step 2? Why this is not applicable to Google/Apple?

There are definitely sanctions against Russia, yet Apple/Play stores work just fine.

grishka10• 2 years ago

For example, Apple store is working in Russia.

Apple hasn't officially sold any hardware in Russia in the last 2+ years. Any Apple devices you can buy come from "parallel import" and are priced 1.5x compared to other countries.

As far as I know, the only way you can pay on the app store is from your prepaid balance at some carriers. Play store doesn't accept payments at all, it pops up a modal saying "payments in Russia are paused".

I can't understand what these sanctions are intended to achieve either. They just make us angrier because there's nothing we can do besides wait it out.

sam_lowry_3• 2 years ago

They just make us angrier because there's nothing we can do besides wait it out.

You are not angry enough.

Get angrier, go out, kill the highest sitting official you can get your hands on, then be beaten, jailed, signed up to war where you will die and receive a posthumous medal?

TiredOfLife• 2 years ago

Not only is Apple store working in Russia. Apple, who supposedly has stopped selling hardware there, works closely with russian government to remove apps like VPNs from the store.

betaby2• 2 years ago

So how that works?

The same sanctions:

- CloudFlare - block IPs ( why? what part of sanctions says that )

- Apple / Google - do nothing

I genuinely want to know.

reassess_blind• 2 years ago

I was recently reviewing my Google account session history and saw an active session from some small town in western China. Obviously freaked out, rolled all passwords, spent hours scouring what they could’ve had access to, etc.

Only for the next day, when Google updates the exact same sessions location to my exact real location on another continent.

Google of course won’t show the IP address of sessions anymore, just the “location” so there was no way of confirming beforehand.

bytenaija• 2 years ago

This seems to be more than a year ago. Why is it suddenly trending now?

auguzanellato• 2 years ago

That probably explains the issues I’m having sometimes when pulling images from Elastic registry on hetzner boxes. At least now I know the reason behind that

eli-talktalk• 2 years ago

Cloudflare misidentifies Hetzner IPs as being located in Iran

Related Stories