Geolocating requests with Google Load Balancer for free

https://blog.doit-intl.com/farewell-maxmind-geolocating-requests-with-google-load-balancer-for-free-7938f5adbef0

by vadimska • 8 years ago

199 68 8 years ago

blog.doit-intl.com

trunet• 8 years ago

finally!

misterbowfinger• 8 years ago

Using this:

https://cloud.google.com/compute/docs/load-balancing/http/ba...

The load balancer expands variables to empty strings when it cannot determine their values, for example for geographic location variables when the IP address’s location is unknown, or for TLS parameters when TLS is not in use.

Geographic values (regions, subdivisions, and cities) are estimates based on the client’s IP address. From time to time, we update the data that provides these values in order to improve accuracy and to reflect geographic and political changes.

manigandham2• 8 years ago

If using the CLI, don't put the single quotes or the whole thing will become the header without a value.

vadimska10• 8 years ago

I have used the CLI and the quotes worked fine for me. Here is the complete command I have used:

gcloud beta compute backend-services update app --custom-request-header 'X-Client-Geo-Location:{client_region},{client_region_subdivision},{client_city}' --custom-request-header 'X-Client-Geo-Region:{client_region_subdivision}' --custom-request-header 'X-Client-Geo-LatLong:{client_city_lat_long}' --custom-request-header 'X-Client-TLS-Version:{tls_version}' --custom-request-header 'X-Client-TLS-Chiper:{tls_cipher_suite}' --custom-request-header 'X-Client-Hostname:{tls_sni_hostname}' --custom-request-header 'X-Client-RTT:{client_rtt_msec}'

lightwin2• 8 years ago

Just realized that Cloudflare also provides IP Geolocation, even for free plan.[1] However it only provides ISO country code of the visitors.

[1] https://support.cloudflare.com/hc/en-us/articles/200168236-W...

therealmarv2• 8 years ago

That's great. Do you know if this also works on all cases when I use Cloudflare only as DNS? Edit: After some thinking this cannot work when Cloudflare is used as DNS only.

ivanakimov• 8 years ago

Anything like this for AWS?

lightwin2• 8 years ago

AWS added geolocation to cloudfront in 2014.

https://aws.amazon.com/about-aws/whats-new/2014/06/26/amazon...

sheeshkebab• 8 years ago

AWS cloudfront feature is incomplete - it returns only country code.

Googles and maxmind features include city/lat/lng.

vadimska10• 8 years ago

Not everything goes thru CloudFront. I hope ELB/ALB will have this feature soon too!

iampims• 8 years ago

AppEngine has had this feature for years and it’s extremely convenient!

vadimska10• 8 years ago

Exactly! I was using GAE for 9 years now and it's so easy to have these headers automagically being attached to your requests. Finally, Google Compute Engine has the same convenient way of geolocating requests.

SteveCoast3• 8 years ago

I've been working on and off on a system to build this data for free - https://www.open-geo-ip.com/

It uses your browser location to build the data, bootstrapped with Mechanical Turk workers around the world. It has one feedback loop, where if you use the JS library it will give you the users location if they deny browser location (if we have the data...) and if they do share location, then we update the database (but throw away the last part of their IP address).

There are two or three other loops where we could build more data (which you can get here https://www.open-geo-ip.com/data/download ).

I'm probably going to do a small kickstarter to pay for some cleanup and expansion work or just kill it if that fails to get funded. Any ideas appreciated on things to do with it.

reaperducer5• 8 years ago

it will give you the users location if they deny browser location

Seriously? Did you help build BonziBuddy, too?

SteveCoast3• 8 years ago

Ha.

Well right now a) we most likely have no idea where an ip address is and b) even if we do, it's nearest-city accuracy.... which is exactly what everyone else does.

Denying browser location doesn't stop the app or website from using your ip address against some third party service to figure out where you are.

cpeterso• 8 years ago

Cool project! I've been thinking about how one could build a crowdsourced geoIP database. Bootstrapping with Mechanical Turk is an interesting approach, especially if you can request workers in different cities and countries. One idea I had was a browser extension that volunteers could install to automatically submit their current location and IP address on browser startup.

It would be helpful if the home page demonstrated the passive geolocation results. The page asked for my browser location but didn't show the result.

btw the map zooming is very slow in my mobile browser for some reason.

forgot-my-pw• 8 years ago

Whelp, all those markers almost crashed my browser.

SteveCoast3• 8 years ago

Yeah it worked great when we had 100 data points. Another thing to fix :-)

avivl• 8 years ago

Been waiting for this for a while!

zvirog2• 8 years ago

Finally, i am waiting for something like that for some time.

abomusa• 8 years ago

Awesome, save some efforts of configure ngnix to work with geoip databases.

vadimska10• 8 years ago

yep. also, spares some compute cycles for your app

zvirog2• 8 years ago

Finally, i am waiting for something like that for sometime.

shayale• 8 years ago

At last

brightball• 8 years ago

One thing I will say for Maxmind is that their API was exceptionally good at picking up anonymous proxies like Tor.

I dealt with a ton of fraud a couple of jobs ago and being able to eliminate credit card transactions for people on Tor was huge in cutting down on charge backs from fake card use. The minFraud API was really beneficial as well.

Don't sell the offering short.

vadimska10• 8 years ago

Well, I am sure MaxMind provides a lot of value for many use-cases. However, I just need something very basic (country, city, lat and long) and getting this "out of the box" from Google's LB is blessing for me.

dylz• 8 years ago

Tor is a special case here: you can use any Tor DNSBL and it will pick up 100% of the nodes, effectively.

All Tor IPs specifically and publicly advertise the fact that they are an open proxy.

michaelmior• 8 years ago

I don't doubt that there are a lot of baddies using Tor (and I wouldn't be surprised if that was the majority of Tor users in some cases). It's a shame though that cutting off Tor access is the solution. A very reasonable one from a business standpoint, but unfortunate.

dmoy• 8 years ago

Doesn't have to be a majority of Tor users, if the tiny minority of actors are flooding tons of stolen cc transactions. A normal person may make an online purchase a day or so, while someone trying to get as much value as possible out of a bunch of stolen cc may just spam a lot of transactions.

For the server seeing the incoming cc purchase requests, it's still majority fraud...

ehsankia• 8 years ago

The author does specify that for his own needs, which was basically adding 3 headers to requests, the offering was not worth it. He also does explicitly say that MaxMind is very good. So I don't think he was unfair at all in the post.

dani-doit• 8 years ago

No need for external services from now on.

alexg84• 8 years ago

Great article! It's about time that Google will come with a geo solution

tpetry• 8 years ago

Does anybody know how good the data is? I mean everyone of the geolocation lookup providers has faults and choosing the majority from different sources will have the best results. But maybe Google has much more accurate data?

vadimska10• 8 years ago

Check this out: https://twitter.com/jasoncartwright/status/98882315371465113...

Theodores• 8 years ago

If you are hosting outside of the Google eco-system and want to be able to do geo-IP for a store locator etc., or just to set a cookie on a page with the lat/lon, how does one build a minimal app in the Google cloud to do just this bit of the task?

If this is easy doable then that would be an easy migration route.

vadimska10• 8 years ago

It would be pretty easy to build. Probably something you can do end-to-end in one or two days.

therealmarv2• 8 years ago

see this other comment for full source code: https://news.ycombinator.com/item?id=16915184

forcer• 8 years ago

has anyone tried to compare geolocation accuracy of this solution vs Maxmind? I would assume Country-level will be very comparable (its easy to do), but doing city level accuracy is not trivial

vadimska10• 8 years ago

https://twitter.com/jasoncartwright/status/98882315371465113...

yoram_benyaacov• 8 years ago

Finally we can have it as a not only on app engine :-)

Dragoneous• 8 years ago

Excellent! Another great capability added! This will save us some over head!

sudhirj2• 8 years ago

For those not running on GCE, I run a free service at https://blip.runway7.net/ that piggybacks on the Google App Engine headers. Suitable for calling from the client browser / device, it allows you to ask for location specific resources straight from the client without having to do anything on the server at all.

Code at https://github.com/runway7/blip

sudhirj2• 8 years ago

This is especially useful in single page apps, or static sites, if that's not obvious.

yoavtco• 8 years ago

Wow, that's so cool! I wish my workload was running on Google Cloud

manigandham2• 8 years ago

Note that if you're using this load balancer behind a CDN then the results can be incorrect, although probably generally within the same region.

Also MaxMind does have completely free databases down to the city level, just with lower accuracy compared to their paid products: https://dev.maxmind.com/geoip/geoip2/geolite2/

vadimska10• 8 years ago

This probably will depend on CDN. Google's CDN is part of the Load Balancer itself and therefore it won't affect geolocation. With "3rd party" CDNs it's different, of course.

Finally, the free version of MaxMind is great. Thanks for mentioning it.

htk_• 8 years ago

Wow, such nice developers! offering more user private data to google huh

tzury• 8 years ago

nobody's offering Google anything, rather, Google is offering developers meta data for HTTP requests.

psanchez2• 8 years ago

Nice! Any clue if there is a way this can be enabled automatically on balancers created using kubernetes (on GKE)? That would be really awesome.

vadimska10• 8 years ago

My thoughts exactly. I am now working on this problem myself and I will share my results soon.

psanchez2• 8 years ago

Looking forward!

ne01• 8 years ago

I have always said that the GCP HTTP(S) load balancer is the hidden gem in cloud. I'm glad it's becoming more and more popular (also feature rich).

One feature that must be mentioned is that it allows you to cache content (could be anything: HTML, JPEG and more) for as little as 1 second! CloudFlare requires an enterprise plan and even then you cannot set a TTL lower than 30 seconds.

Shani892• 8 years ago

Finally!

Shani892• 8 years ago

Finally!! Thanks for sharing

neter145• 8 years ago

Sharing is caring :) Thank you

geofocus• 8 years ago

Cool!! Location-based traffic insights have significant value in many use-cases. Show the results on a heat-map to see where you should be targeting customers.

databill2• 8 years ago

We've done some testing and found many inaccuracy in Google LB Geo-IP results, anyone know who is in charge and which channel we can feedback to Google?

databill2• 8 years ago

or anyone would like to try our service, please drop me a mail data at ipip.net

Geolocating requests with Google Load Balancer for free

Related Stories