How Referral Spam Like lifehacker.com Gets into Google Analytics

http://kraftblick.com/blog/referral-spam-in-google-analytics/

by NonMathGirl • 10 years ago

185 50 10 years ago

faitswulff2• 10 years ago

"Error establishing a database connection"

Looks like HN traffic took it down. Here's Google's cached verson (Archive.org didn't have it): https://webcache.googleusercontent.com/search?q=cache:cC67C6...

zhego• 10 years ago

Their website is live again http://kraftblick.com/blog/referral-spam-in-google-analytics...

nodesocket2• 10 years ago

I had also gotten absolutely swamped in Google Analytics with things like:

  Language:
    Secret.ɢoogle.com You are invited! Enter only with this ticket URL. Copy it. Vote for Trump!
    o-o-8-o-o.com search shell is much better than google!
    Vitaly rules google *:｡゜ﾟ･*ヽ(^ᴗ^)ﾉ*･゜ﾟ｡:* ¯\_(ツ)_/¯(ಠ益ಠ)(ಥ‿ಥ)(ʘ‿ʘ)ლ(ಠ_ಠლ)( ͡° ͜ʖ ͡°)ヽ(ﾟДﾟ)ﾉʕ•̫͡•ʔᶘ ᵒᴥᵒᶅ(=^ ^=)oO
    Google officially recommends o-o-8-o-o.com search shell!

However, I recently applied the following account level filter and seems to have helped a ton. I can't guarantee this will 100% work, but it has certainly helped.

http://imgur.com/a/z9gSg

EDIT: Here is the regex so you can copy and paste it:

  \s[^\s]*\s|.{15,}|\.|,

chinathrow2• 10 years ago

I see the same spam. At this point I think the filtering by the average GA user base costs Google way more CPU cycles than an overall firewall ruleset.

nodesocket2• 10 years ago

Wait a day. The filter only affects traffic after it was applied.

faitswulff2• 10 years ago

Note that the "k" is a Cyrillic "к" and the trick is to get analytics users to click referral spam:

Referral (or ghost) spam wasn’t that innocent. Curious marketers and web analysts checked domains they supposedly got traffic from. Referrers got them transferred to trashy websites with ads, viruses or porn.
...[Vitaly] needed software to get his websites into people’s analytics reports. His words were “if only one out of 1000 people click the link to see who these referrers are, I’ll gain profit.”

I just installed Google Analytics on my site and saw these weird referrers, and my site is quite obscure. Seems like this could be a widespread problem.

There's also a fix in the article from Georgi Georgiev: http://blog.analytics-toolkit.com/2016/language-spam-latest-...

Log in to your Google Analytics account and navigate to Admin -> Filters area. Add a new filter with the following settings. Make sure you have the “Edit” access at the “Account” level in Google Analytics. Remember that the filter will eliminate traffic hits where the language dimension contains 15 or more symbols.
Filter Pattern is .{15,}|\s[^\s]*\s|\.|,|\!|\/.

soared3• 10 years ago

Any guesses on the value of the "google.com" with the Cyrillic G? I'd say $1MM.

notatoad• 10 years ago

i thought that when icann allowed non-latin characters in URLs, they were going to limit it to only the ccTLD where those characters were in common usage - so ԍooԍle.ru would be valid, but ԍooԍle.com wouldn't be registerable.

am i misremembering, or did that plan change? this just seems like a phishing nightmare.

DenisM• 10 years ago

There's also Cyrillic o / о and e / е. And l can be substituted for the capital I or the digit 1. We're looking at quite a few different fake "google" domains. I doubt any of them will be worth a million bucks, they are not that scarce unless someone registeres all of them at once to squeeze the market.

mutatio• 10 years ago

Probably < $0. It's a deceptive domain of a strongly contested trademark - all the Unicode variants are probably just burner domains.

csydas• 10 years ago

Probably not much cause it'd look like гоогle.com or something else silly. The Cyrillic g character would stand out and most Russians would go to yandex before Google so accidental Russian traffic would be barely worth the effort. The print form of cyrillic that would be used in browsers just doesn't have a character to make the domain worth anything.

The vowels on the other hand look the same. ОO, eе, аa.

newman314• 10 years ago

Couple more links that I have found useful.

* https://moz.com/blog/how-to-stop-spam-bots-from-ruining-your...

* https://megalytic.com/blog/how-to-filter-out-fake-referrals-...

* http://help.analyticsedge.com/spam-filter/definitive-guide-t...

ifrins• 10 years ago

I also receive all sorts of spam hits on Google Analytics accounts. My suggestion is to create a filter that only includes the traffic directed to your hostname. It filters pretty much everything. But it's another problem in mobile apps tracking. I've yet to find a solution on mobile. Any suggestions?

t3ra• 10 years ago

Isn't cloudflare blocking this spam? Because I was managing a WordPress site for a client behind cloudflare and they were still able to spam usi g this technique

hbcondo714• 10 years ago

No, the referal spam never hits the website. It instead goes straight to GA.

soared3• 10 years ago

hbcondo714 is correct, nothing every actually touches your website. Its send with the measurement protocol, which allows you to POST data straight to GA.

https://developers.google.com/analytics/devguides/collection...

klenwell• 10 years ago

This is the detail that the article seemed to miss. Or did I miss it?

partiallypro• 10 years ago

Cloudflare does block some spammers, and marks them as "bad browser." But it doesn't catch all types of spam attacks.

https://support.cloudflare.com/hc/en-us/articles/200170086-W...

shortformblog2• 10 years ago

This has been written about a few places, including Motherboard (where, full disclosure, I contribute):

https://motherboard.vice.com/read/this-pro-trump-russian-is-...

Not long after doing the interview with Vitaly, they had to write a follow-up because the original article ended up being used as GA spam:

http://motherboard.vice.com/read/spammer-now-spamming-google...

That cycle has repeated itself a few times (The Next Web ran into something similar a month prior).

To be honest, the real problem here, far as I can tell, is that Google hasn't figured out a way to make whitelisting traffic easy in Analytics.

Most people don't use a tenth of GA's overall power, and as a result have no idea where to look when it comes to building out filters and such. Yes, those who seriously do analytics for a living know how to handle this, but the problem is, there are a lot of people who use GA because it "just works," and it doesn't do the job if someone's just throwing junk into the referrers.

This problem is widespread and it depreciates the value of a fundamental Google product. Why is a company that's so good at filtering spam apparently ignoring the problem in this case?

thewhitetulip2• 10 years ago

Exactly, I started using GA a few months back, have no clue how to filter this out.

marktangotango• 10 years ago

Add a filter that only allows data from your domain. I found this on a spammy content marketing site somewhere. I added this and haven't seen any referral spam since. I'm not sure what the pros and cons are, maybe someone here can elucidate?

In GA: Admin -> All Filters -> Add Filter -> Predefined Filter Type -> Include Only -> Traffic to the hostname -> tha contains -> Enter your hostname in the text box ie mysite.com.

bhartzer2• 10 years ago

Google Analytics already filters out fair amount of bot traffic from Google Analytics (compare your server log files with GA traffic and you'll see what I mean). Why they haven't tackled this problem is a mystery.

enzanki_ars• 10 years ago

The traffic mismatch is not due to any filtering by Google. Most people useing some sort of ad-blocking technology have Google Analytics blocked by default, preventing any traffic info being reported.

soci• 10 years ago

I found a guide to get rid of all referral and language spam in GAnalytics. So far, works perfect for me. It filters out past spammy data and also adds a new View to start tracking data without all the spam. https://www.ohow.co/ultimate-guide-to-removing-irrelevant-tr...

codedokode• 10 years ago

I do not understand how domains like ɢoogle (with some unicode G-like character) are registered? I remember when IDN were introduced there were concerns about spoofing similar URLs and there were some restrictions on what characters are allowed.

So now the owner of ɢoogle can add a signup form and start phishing for Google Accounts?

And I am sure Mr. Trump has no relation to this spam. Obviously it is done to turn people against him.

soared3• 10 years ago

I work at a digital marketing agency and recently had to deal with this. (I also wrote two blog posts on it, currently unpublished). I have access to ~50 properties in google analytics from work and personal accounts. About 35 of them were affected. I couldn't discern a pattern on size or type of website. Very interesting how widespread this was, but very very easy to filter out. It takes 2 seconds to filter this type of spam from appearing.

A side note though, the author is incorrect on point #3 of how he expects future spam to behave. This spam is not sent with bots clicking on links acting like user. Spammers use the measurement protocol [1] to post data to randomly-ish generated universal analytics ids. So they take that referral/utm data and continuously post it to random UA-IDs until they are successful, then hit that ID much harder.

Previously you could get around this by making your first property (GA structure is a tree, account > property > view, where you can have multiple properties and views) a shell. You'd never use the first property and only the second or third because then your UA-ID goes from 5575393 to 5575393-2. That "-2" previously wasn't attacked.

[1] https://developers.google.com/analytics/devguides/collection...

alxmdev• 10 years ago

Lifehacкer.com mimicked lifehacker.com with the only difference in Cyrillic letter ‘к’ instead of Latin ‘k’ used in the original traffic source. The substitution was obscure.

The Wikipedia article "IDN homograph attack" describes various ways in which web browsers and ICANN try to protect users against this sort of shenanigan: https://en.wikipedia.org/wiki/IDN_homograph_attack#Defending...

cypherpunks01• 10 years ago

Why was the HN title moderated (presumably) from lifehacкer.com to lifehacker.com?

Sir_Cmpwn• 10 years ago

The title is normalized automatically, I think, to prevent phishing or something.

chinathrow2• 10 years ago

I deal with this Vitaly on a daily base.

Google failed to block him for months - it would be so easy though. He hits / on my sites but we never ever serve Google Analytics in this URL, only from /<lang>. Hence all IPs logging pageviews on / are 100% spam. No false positives.

thewhitetulip2• 10 years ago

Yes, the last time I have his spam is Dec 23, and it changes the text, the o-o-o-h shell and vitaly etc etc Google's failure to block the spam is huge considering all the money they make is via adverts and having GA is critical to their $$$

bhouston• 10 years ago

Can not google just figure out this is analytics spam?

I suspect a similar filter to like what determines spam email can be used? Basically a bunch of rules would look for similar referrals popping up all over the place and are invalid.

driverdan• 10 years ago

Poor user behavior metrics in Google Analytics negatively affected website ranking in Google organic search results.

Is this actually true? Does anyone have a primary source for this?

angry-hacker• 10 years ago

No it's not true because this referral spam never touches your server. Google does measure bounce rate when you come from Google search results, but this is entirely different scenario.

jffry• 10 years ago

I was reading along, about four paragraphs in, and then I was accosted by a modal dialog: "Get Your Copy of "AdWords for B2B""

Maybe your article is useful, maybe not, I don't know because I closed the tab, but screw you and your asshole design[1]

edit: No seriously, take a step back and look at your website in a mirror [2]. Popping up _another_ nag as I keep scrolling down? Lord almighty, you've motivated me to blackhole your domain in /etc/hosts

[1] https://www.reddit.com/r/assholedesign/ [2] http://imgur.com/a/Hgq9S

herbst2• 10 years ago

She is also spamming reddit badly and then answers with things like "this is not spam" "i am a girl btw". I blacklisted the site.

NonMathGirl2• 10 years ago

It looks like you have troubles with me. I apologize if I made you feel insecure and victimized. Sorry again, dude.

aakarpost• 10 years ago

If you want to clean up your historical data in Analytics, you can create a segment and block the language spam.

\s[^s]*\s|.{15,}|\.|,

Go to Audience Overview > Add Segment > New Segment > Conditions > Select Language > Select Matches Regex & enter the above regex. Select Session & Exclude, give it a name and hit save. Congratulations, now your historical data is clean.

snowwrestler• 10 years ago

I have been able to thoroughly block referrer spam by simply setting a custom segment that only passes traffic that exactly matches my site's hostname. Does this not work for everyone?

EDIT: I see some comments with advice to set a regex filter. BE CAREFUL WITH GA FILTERS. They discard traffic permanently! If you misconfigure one, you could lose real traffic data and never get it back.

This is why I generally prefer to use custom segments, which hide but do not discard traffic. However, I have to remember to set the segment(s) manually.

If you want to use filters, create a new view and set the filter there. Keep your original view as a backup.

Best practice with GA is to create two views when you first set it up. Call one "All Data" and don't touch it. Call the other something like "Reporting" and set filters etc there. That way you always have a "backup" set of data.

ericdykstra• 10 years ago

I use Clicky as my default analytics tool for a new site. It filters out referral spam automatically which, by itself, justifies the $10/mo for the basic plan (which covers multiple sites).

http://clicky.com or http://clicky.com/100950546 if you want to use my referral link.

Referral spam is just a waste of time, and if a site like Clicky can fight it effectively, I don't see why my Google Analytics is constantly littered with it. Gmail's spam filter is amazing, but their referral spam filter seems non-existent.

revelation• 10 years ago

So someone is sending fake referrers that get into peoples favorite hidden data collection tool, and these people have summarily complained here. And I'm supposed to care?

Who knew, vacuuming up data from your visitors may not be all its made up to be. Someone should hand this guy a medal. I should look and figure out if browser extensions can manipulate the referrer yet, spamming all those trackers and analytics services with fake data seems like a wonderful idea.

akerro• 10 years ago

They don't get into Piwik. https://piwik.org/

bhartzer2• 10 years ago

Sure, we can filter it out, but the real question is, “when will Google take care of referral spam?”. Referral spam has been plaguing Google Analytics for a few years now.

For some websites (with less than 100 visitors a day) it’s gotten so bad that it’s almost making Google Analytics completely useless now.

kevin_thibedeau• 10 years ago

I've dropped GA and all other Google services from my sites specifically because of their inaction in fixing this problem. They know what the spam domains are. They should be filtered automatically.

jgalt212• 10 years ago

Unicode domain names are just a bad idea.

herbst2• 10 years ago

Ih god. Not only are you spamming reddit but also here.

(Sorry guys for the unfriendly comment but the same name on reddit is basically a spam my articles account)

NonMathGirl2• 10 years ago

Your persistence in leaving comments here and there for my humble blog post is incredible!

How Referral Spam Like lifehacker.com Gets into Google Analytics

Related Stories