What to do when your site is "too large" for Google Analytics?

http://webmasters.stackexchange.com/questions/2141/site-too-large-to-officially-use-google-analytics

by mwsherman • 16 years ago

46 33 16 years ago

webmasters.stackexchange.com

jacquesm3• 16 years ago

I got the exact same email today, and had to read their terms of service to know for sure I wasn't being had.

Apparently you need an adwords account 'in good standing', I'm not sure what they mean by that because we do have an adwords account, and as far as I know and can see it is in 'good standing', only we're not advertising any more using google because we found that it isn't making enough money in return.

See:

http://news.ycombinator.com/item?id=1593898

I'd pay for analytics, it's a very useful service and I can't see the adwords angle, after all with something as vague as 'good standing' as the deciding factor how about simply charging for analytics?

I always figured that analytics was free because it provides google with useful data as well.

Maybe it's time to get my old stats suite dusted off and re-installed, I don't like operating against the TOS even if it is apparently something they'll let you get away with.

tibbon• 16 years ago

Good standing doesn't seem to be vague to me really. Your account isn't violating some ToS, delinquent, etc... Most accounts are probably in good standing.

I wouldn't worry about this too much. It doesn't cut you off, just limits it to updating once per day (basically just a day behind). Its possible that your business/company is dependent on information from hourly reporting in realtime, and if that's the case you should probably augment GA with some other paid service or roll your own BI internally.

jacquesm3• 16 years ago

I think it is a bit weird (and points to a fairly serious lack of 'Chinese walls') that google would change the behavior of their analytics package based on your account for a completely unrelated offering.

I see adwords, adsense and analytics (as well as search) as unrelated offerings by google.

1248165• 16 years ago

> I'd pay for analytics...

You could just take how much you'd pay for analytics and buy some ads. :-)

lmkg2• 16 years ago

Professional Google-certified web analyst here.

According to the terms of use I'm familiar with, linking an AdWords account to your GA account, and spending money on it (even nominal sums like a buck fifty) has been sufficient to be exempt from any hit limits on GA. Your AdWords account does need to be explicitly linked to the GA account, and you do need to be spending some non-zero amount of money, but they're easy to connect and that's honestly still quite cheap for an analytics implementation. Omniture SiteCatalyst will run you in the six figures per year, minimum. It's possible that Google has changed its policy, as Google has a reputation for being somewhat fickle as an analytics provider, but I haven't heard anything via official channels.

That aside, only having your data update once a day should not be a problem. There is surprisingly little value in data updating multiple times per day. There's no reason to have data update more frequently than you can take action upon it, and most sites of most companies move at the speed of a glacier. There are a few business models where it is beneficial, for example news sites can benefit from knowing what stories are really popular right this minute. Besides, if you need data more often than once a day, you shouldn't be on Google Analytics anyways. Average data delay is a few hours, but it's not unheard of for data to be stale by a day or two.

papa• 16 years ago

I've run into problems with hitting a number of limits on Analytics as well and still have no good solution.

A couple of solutions I'm aware of:

1) Omniture Site Catalyst: A lot of big sites use this but it will cost you dearly. My site runs about 85mm pvs/month and they quoted me close to a million dollars a year at those traffic levels! I didn't even bother negotiating it down, even the negotiated price would still have been way more than I'm willing to pay.

2) Urchin: I don't know how similar/dissimilar Urchin is from Analytics, but the Urchin 6 software runs about $3k. This option could prove labor intensive since you would effectively be running and logging your own metrics on your own server. So the actual costs could go up (wrt to time, equipment, etc.)

There are plenty of smaller players in this space, but the problem that always comes up is, "Will these guys be around in a year?". When looking for an business solution, it's nice to know that your data won't up and disappear should the company go bankrupt.

I do wish Google would provide a paid service for those of us that could use it and would pay for it. I'm already using Google Doubleclick for Publishers for ad serving (which costs money), you'd think they'd also want to expand their other complementary offerings (like metrics).

landyman• 16 years ago

Other services I've seen that seem to not fall into that "smaller players" category: Coremetrics, Webtrends, Unica, OneStat, and Clicky. I've worked with all of these (and some smaller players too), and I must say that for the value, you really can't beat Google Analytics. Even Omniture will have some times when the data is not available right away; especially as you start doing some more complex things with it.

naturalized• 16 years ago

We do not want Google to know the true size of our service, so we have analytics code in every 5th pageview (when templates are constructed by scripts, it inserts the analytics code only when the UNIX time is divisible by 5.

dennisgorelik• 16 years ago

What's the reason to hide your size from Google (or anybody for that matter)?

rodh257• 16 years ago

though that would skew the referrals and other metrics wouldn't it?

oasisbob• 16 years ago

If your numbers are large enough, no. As long as the selection isn't biased, it doesn't matter.

Another commenter on this topic has a good post about convincing management of the validity of sampled metrics, I'd read it then tip your chair back and stare at the ceiling for a while.

brown9-2• 16 years ago

Well, now they know.

tcarnell2• 16 years ago

I thought every number is divisible by 5

nl• 16 years ago

Umm - I assume you know this, but inserting the analytics code only when the UNIX time is divisible by 5 isn't the same as every 5th pageview.

(Also, doing it on a pageview basis instead of visitor basis means you lose all the nice visit path tracking Analytics does)

nostromo• 16 years ago

Buy Urchin and run it yourself -- it's virtually identical to Google Analytics (or at least used to be) because it's the company that Google acquired to make GA.

Avoid Omniture SC. It has a terrible interface, is too expensive, harder than GA to implement, and Omniture doesn't seem to be improving it much, if at all anymore.

tcarnell2• 16 years ago

Show off on Hacker News?

On that theme does anybody know what the maximum limit of money is before your bank account crashes? (or you have to start buying hotels instead of houses)? :-)

bl4k• 16 years ago

I once tried to deposit enough money in my bank account to cause a buffer overflow. I could never get the shell code to exec though.

drosenthal• 16 years ago

I designed and built high-end web analytics software for years.

Sampling your data by visitor/cookie (not by page view) is the number one thing to do that that no one does. Just sample your data down by a factor of 10 or 100. The costs drop by a factor of 10 or 100, query speeds improve, and the business value of the data drops by very, very little.

The only unfortunate prerequisite for this approach is that you need a company/boss that understands that sampled numbers are OK even if they are not exact. (So, explain, if you must, that your industry insider source tells you that the notion of 'exact' in web analytics world is very loose indeed.)

pierrefar• 16 years ago

Yep. For queries with large volumes of data, Analytics does do sampling and gives you error margins (+/- x%) next to each number. I found that having these margins helps convince higher ups that it's OK, and also tells me when it's not OK and have to dig a bit more.

1248165• 16 years ago

I think drosenthal is talking about sampling the number of hits you send to analytics. GA's tracking api supports this:

http://code.google.com/apis/analytics/docs/gaJS/gaJSApiBasic...

loup-vaillant4• 16 years ago

Do not use it?

More seriously, what Google Analytics does that you can't do yourself when you have complete access to the logs of your web site?

loup-vaillant4• 16 years ago

My question is serious and genuine. I expected answers, not votes.

Now I recognize that it is a complex question: If Google Analytics has some advantages that you can't reproduce even when you fully control your web server, I would like to know what it is. If it does not, then we may want to stop sending all our logs to Google right now.

jasonkester• 16 years ago

The reason that Google is choking on this 30M pageview/month site is because analytics for a 30M pageview/month site is hard.

Things like unique visitor, unique referer, unique whatever tracking just explode when you have to compare each IP address you get with 20M-entry list.

If you try to do it on your webserver, you'll have a 3 hour window each day where a significant portion of your CPU and Memory is not available for serving web traffic. If you try to do it elsewhere, you get to ship 20gb daily logfiles around.

It's just not any fun. Which is why even Google has difficulty with it.

Running your own analytics is great for relatively small sites. For big ones it starts to bog down once you hit a certain size.

1248165• 16 years ago

Most stats providers compute unique visitors using client-side cookies; not entirely accurate, but quite cheap. You're right though, it's a lot easier to use an existing (and free or nearly free) tool than to build your own.

What to do when your site is "too large" for Google Analytics?

Related Stories