Ask HN: Are you running a web crawler off the following IPs? It's broken

https://news.ycombinator.com/item?id=5332942
by latitude • 13 years ago
22 4 13 years ago

Fellas,

Whoever is running a web crawler on one of the following IPs, check your code. It is sending some garbage as the User-Agent string, e.g.

   RMID=007f010001c2513733c300ea; 
   SERVERID=w2; 
   SS_MID=672bfcf3-7c0e-492c-b4f5-c4006e444f5chdygi1wn;
   ss_lastvisit=1362572526119; 
   hbfec=OePK5KX7vFq7S0Uk-9GxBTrsv9aAR0LR8BxQl+rkdBh+nV56OyysSNYKFbLQ-; 
   gae_b_id=X2dhZV9iaW5nb19yYW5kb206NjFDcDJxVzZNRGxadHNGdEFmaTVOZ25...
   GOOGAPPUID=582;
   BBC-UID=556113279384de3e4f43d838f1fec2a6700c2f5db484512e9aa1244e...
   ...
Over 4KB of what appears to be some internal crawler's state or data. Involved IPs are:
   107.22.13.225        (amazonaws.com)
   207.97.227.247       (aux1-ext.rs.github.com)
   46.51.203.140        (amazonaws.com)
   54.235.37.145        (amazonaws.com)
   75.101.190.67        (amazonaws.com)
The crawler is accessing only one page on my server, the one linked from the HN's front page, so I think it's someone from here.

Related Stories

Loading related stories...

Source preview

news.ycombinator.com