Apple might be running a web crawler written in Go

http://jan.moesen.nu/2014/11/06/apple-crawler.txt

by beltex • 12 years ago

169 83 12 years ago

jan.moesen.nu

jezfromfuture• 12 years ago

Way to get someone sacked at apple.

vetinari• 12 years ago

Why? This is exactly the situation, where the tool is a great match for a job.

mseepgood• 12 years ago

They're looking for Go engineers: http://www.golangprojects.com/golang-go-job-in-Senior-CDN-De...

andreasklinger• 12 years ago

Why shouldnt someone at apple use Go.

Servers/scripts/etc are not their usual focus and they dont really have an internal language that fits better. Go is pretty good at this kind of things.

pjmlp5• 12 years ago

I like Go, but fail to see how Swift and C++14 don't fit the bill as well.

EDIT: changed to a more constructive comment.

aikah• 12 years ago

lol,i'm not sure why anybody would be sacked because "Google product" ;) however,I wonder why they didnt use Swift,and it's a question.

wyager4• 12 years ago

Swift was designed for coding user-facing applications, not internet utilities.

melling3• 12 years ago

It's not as mature. Doesn't have the library and third party support. Does Apple use Mac OS for all its servers? They use Azure for some servers? That would imply they need the language to be cross-platform.

kaolinite• 12 years ago

The most likely explanation for not using Swift is the employees who worked on the crawler simply didn't know it existed. From what I've heard, only a select few at the company knew it was in development and a number of Apple employees were noticeably shocked during the WWDC keynote when it was announced.

Apple actually uses a bunch of 3rd party developer tech on the server-side and on the web. They use Rails quite a bit I believe and are starting to use Angular.js too.

132• 12 years ago

I've got several thousand requests for this in my logs with an IP address in the same /24.

Two weeks ago it was using a different user agent:

    Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Fetcher/0.1)

This string is mostly from a beta build of iOS 6.0 beta 4, but with the same suffix the author discovered.

unwind2• 12 years ago

Not at all sure why it's relevant which programming languages they use at Apple.

Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what?

The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah.

EDIT: Mis-typed the netmask, I meant /8 but typed /24. Fixed.

manacit• 12 years ago

A /24 for a company of their size is nothing - they actually have the entire 17.0.0.0/8 to themselves: http://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addres...

jrochkind1• 12 years ago

Less about 'of their size' then about how long they've been around, since back in the days when a large company could get a /8, and just about anyone could get a /16.

unwind2• 12 years ago

Yeah, that was what I meant, brain error. Thanks, I edited my comment.

runeks• 12 years ago

Interesting. I know Google uses the IP 8.8.8.8 (and 8.8.4.4) for their public DNS servers. So it seems Level3 doesn't own the entire 8.0.0.0/8 block then.

wyager4• 12 years ago

The takeaway for me was that Apple has a whole /24 subnet to themselves. That's just ... immense, for a single company. Gaah

(I think you mean /8.) Brace yourself: http://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addres...

hueving• 12 years ago

A /24 is only 255 addresses...

mchanson• 12 years ago

Back around 1990 I had a class /16 and I was just a 14 year old. Before the web explosion it was a different time. Domains used to be free back then too!

sspiff• 12 years ago

I was still playing with toy cars back then, but how did people lose those IPs? Were they revoked?

fuckballs• 12 years ago

Eeek, the "C" in CIDR stands for classless, so it is either a Class B or a /16 not a "class /16"

Also, it is highly unlikely that you had a /16 as a PI allocation even in 1990.

icedchai• 12 years ago

I still have a /24 from the era.

chatmasta• 12 years ago

You might find this guide to calculating subnets helpful. ;)

http://www.dslreports.com/faq/cisco/30.1_Quick_subnet_calcul...

untog• 12 years ago

People are curious simply because Apple is so damn secretive. I'm not sure I'd ever want to work there because it seems I would be unable to talk to anyone about my projects, ever.

dagw• 12 years ago

HP has two /8 subnets. What's even weirder is that Eli Lilly (the pharmaceutical company) and Halliburton (the oil company) also both have one.

(as others have said, you mean /8)

jackhammer• 12 years ago

I seems like organisations that where early adopters of the internet got one.

yebyen2• 12 years ago

Maybe the poster expected Apple to eat their own dogfood? They have ObjC and Swift, and Google is one of their major competitors, in mobile space yes as well as pop-language space.

A /24 is also only 256 addresses, so really unremarkable. I think you are reading it backwards. MIT has an /8, that's the large one.

yebyen2• 12 years ago

Mea culpa, I see that Apple does indeed have an /8.

Bahamut• 12 years ago

The software tech Apple uses generally doesn't seem to get as publicized compared to most large tech companies - for example, I know second hand that Apple uses Angular.js, but I have never seen anything in the wild about it.

One thing to keep in mind though is that Apple is much more a hardware company than software.

NateDad4• 12 years ago

For a hardware company, the write a crapload of software... like Mac OS, iOS, and all the built-in applications. I'd say they probably do a lot more on software than hardware, actually. They only release a few pieces of hardware a year.

beltex• 12 years ago

"Apple views itself as a software company."

- Steve Jobs

https://www.youtube.com/watch?v=dEeyaAUCyZs

NateDad4• 12 years ago

Go is not really a "Google" technology. It's just a programming language like any other. There's no reason Apple shouldn't use it, if it's the right tool for the job.

josu• 12 years ago

Well, it'd be pretty weird if Go-ogle started developing their applications in Swift.

sandipc• 12 years ago

I agree. "Breaking news: Apple uses Google products"

wsc981• 12 years ago

Relevant XKCD comic "Map of the internet": http://xkcd.com/195/

hartator• 12 years ago

I am doubtfull Apple wants to compete with Google in search.

Or maybe it's a way for Steve Jobs to get a revenge post-mortem on Google for Android!

yourad_io• 12 years ago

So far, I have seen requests from two IPs: 17.147.18.33 (7 on 2014-10-15) and 17.147.18.33 (7 on 2014-10-15)

One of these should be .35, judging by the logs that follow ;)

struct• 12 years ago

Perhaps a proxy designed for iOS testing?

132• 12 years ago

I don't think so, it's only requesting HTML.

pkulak• 12 years ago

Go is a fantastic language for writing a crawler. I wrote the Showyou crawler in Go and it's both one of our highest load processes (and it's very efficient, trust me) and most stable.

brycekahle• 12 years ago

The Go http client definitely has a bug that doesn't maintain the User-Agent across redirected requests.

https://code.google.com/p/go/source/browse/src/net/http/clie...

bgentry• 12 years ago

Looks like issue #4800 is the place where it's tracked:

https://code.google.com/p/go/issues/detail?id=4800

jdalgetty• 12 years ago

I've been seeing this as well.

bkeroack• 12 years ago

I've seen golang mentioned in Apple job listings. I didn't think this was a secret.

christop• 12 years ago

I noticed this as well a couple of weeks ago.

They're still actively visiting every page on the websites that are associated with our iOS apps.

Today alone (starting shortly before 8am CET) they've crawled over 8000 pages on https://trails.io and http://offmaps.com — without sending conditional caching headers (our pages don't change too often).

    $ grep ^17\\. /var/log/nginx/*.log | grep 'Go|Fetcher' | wc -l
    8254

Note that they don't appear to be scraping any URLs available from within the apps themselves, but rather the the company/support websites linked in our App Store listings.

I guess they're automatically scanning for objectionable content, since these websites are linked from the App Store and the iTunes website?

0x02• 12 years ago

Ooh, that's interesting. They already increased the requirements for online privacy policies etc for ios8 apps, so perhaps we'll see some crackdown on appstore apps who are missing those.

0x02• 12 years ago

So they are "apple-maps'ing" search as well, hilariously using one of Google's own tools in the process.

Would be interesting to know if/when any deals with bing/google/duckduckgo for search is expiring, like they were with google maps. That's probably when we will see an apple search engine.

Makes sense for them to fully control another part of the backend for spotlight and siri.

pvsnp• 12 years ago

It looks more likely that someone's learning Go and going through the crawler example in the tutorials. Of course I may be wrong here. The trigger for me was the term "Fetcher" here.. http://tour.golang.org/#73

Fun exercise though.

kngspook• 12 years ago

As a former employee, that was my first thought when I read the post too. Every once in a while, I would stay late there and work on a little personal project for fun there. Sometimes one of my friends would be working late, and I'd just keep him/her company. After all, if I'm just going to do the same thing at home, I might as well take advantage of coding on my big monitor and comfy chair...

pyman• 12 years ago

IMHO, a big couch is better than a comfy chair.

lifeforms• 12 years ago

If Apple coders are reading, I noticed exactly the same issue of User-Agent header not being used on redirects.

You can solve this by (ab?)using the CheckRedirect functionality of http.Client to set the User-Agent again.

Here's an example: https://github.com/lifeforms/httpcheck/commit/07440d952d1660... (The program is nothing special, just a little thing I use internally for monitoring)

sean9999• 12 years ago

This just in: Someone at Apple was doing stuff

Apple might be running a web crawler written in Go

Related Stories