Replace Google Analytics with a shell script

https://www.go350.com/posts/replace-google-analytics-with-a-shell-script/

by _zjfx • 5 years ago

275 105 5 years ago

chalst• 5 years ago

I rather dislike the surveillance capitalism model Google promotes, but the article is a little unfair. As a payload, the snippet of code that calls the GA library is pretty lightweight and there are many ways to make this asynchronous: this really does not need to have a significant impact on user experience.

As an aside, I find Google's model to be less bad than Facebook's attention capitalism model because it permits a diverse, unoptimised and decentralised web, amp notwithstanding.

HWR_14• 5 years ago

I think it's a tad disingenuous to focus only on the initial loading of Google Analytics. Presumably there is some ongoing cost to the user regardless of load time and it's probable that the blog was using the synchronous version. There may be many ways to make this asynchronous, but they chose the easiest way: removing it.

eli4• 5 years ago

The google analytics tags have defaulted to loading asynchronously for many years now.

anonydsfsfs2• 5 years ago

Google Analytics by itself is lightweight, but if you're at a large company with a marketing department, my experience has been that it never stops there. We've got pages with literally dozens of third-party Javascript trackers, and I've had to continually push back against requests from marketing to add more. While each one individually isn't a big performance hit, they add up fast.

Same principle applies to security: each of these trackers has basically full control over the page, and if one of them gets compromised and replaced by malicious code, it would be disastrous. The probability of any individual tracker getting compromised is low, but what you should be worried about is the probability of at least one being compromised, and that probability rises exponentially as you add more.

datastoat• 5 years ago

The probability that at least one tracker is compromised actually grows sub-linearly in the number of trackers.

Let n be the number of trackers, let p be the probability that an individual tracker is compromised, and assume compromises are independent. Then: Prob(at least one comp.) = 1 - (1-p)^n ~~ np. In other words, linear in n. This approximation is fine for small n*p. For large n*p, the probability has to be <= 1 of course, so the real probability is sublinear.

d0gbread• 5 years ago

From my experience this is a function of client side libraries not having an "owner" in place to refactor. Marketing often doesn't know or care about performance, so of course they'll just want to add over time if it makes them more effective at their job.

When businesses invest in caring about this (and it can even be just one headcount for the biggest of businesses) there's opportunity to reign it in and maximize the value from a given platform, reduce it the need to add others. That helps with security, governance, and adoption of what's there.

Nevermind more forward looking solutions like server side which gets these injected scripts off your site completely and greatly improves security.

sneak• 5 years ago

I run a self-hosted analytics tool called Matomo (via CapRover, a selfhosted PaaS) that I include on my static site pages. No third parties are involved unless you count Hetzner where I get my bare metal.

Best of both worlds.

ev1• 5 years ago

I wish Matomo/Piwik worked well on larger sites, or supported Postgres. It seems to start falling apart on my joke sites that get a few million PV/day.

cpach• 5 years ago

Have you tried adding more servers for Matomo?

https://matomo.org/docs/requirements/#tracking-100-million-p...

mraza0072• 5 years ago

This is a very simple and unique approach to gather metrics of your site while maintaining the privacy of the user.

I’m just curious how can you use this solution with a static site generator like jekyll

slimsag7• 5 years ago

The article approach requires access to apache access logs, so you'd need to host your static site on an apache (or other) server with logs you can access.

mraza0072• 5 years ago

Got it makes sense :) This means I can’t use this with github pages

slimsag7• 5 years ago

I just deployed Plausible.io[1] for my blog $6/mo on a droplet, it's self-hosted and 100% open source. It seems very privacy focused, and provides basically the same minimalist data described in this article.

I always find it weird though that privacy advocates still go for "I have this hidden treasure trove of analytics about my site that you cannot access"

Analytics should always be 100% anonymous and aggregated. And if you've met those two bars, making them open and public should be a non-issue. If you can't make your analytics public, that's called spying. There should be no issue making your analytics public. Mine are[2]

[1] https://plausible.io/docs/self-hosting

[2] https://devlog.hexops.com/about#open-analytics-data

kbenson3• 5 years ago

$6/mo on a droplet

I wonder how many extra dollars cumulatively DO makes from the $1 premium on getting a better Intel CPU or AMD CPU and/or NVMe up sell on the smallest droplets.

I pay the extra dollar as well. I doubt I actually see the difference for what I use it for. I feel no desire to pay a dollar less though, so they are making another 20% from me on that product than they would have.

slimsag7• 5 years ago

I didn't pay the extra dollar, I got the $5 droplet - but the backups are $1/mo extra

hk1337• 5 years ago

I would imagine the main thing is not wanting people to or derive confidential company information, like how a certain marketing strategy is going or some other piece of data you can get from analytics.

ccvannorman• 5 years ago

OK, I find it hilarious that your open analytics say zero hits for the past few weeks until 70+ hits from today because you linked your open analytics on an HN comment... :-D :-D

slimsag7• 5 years ago

I only deployed it just earlier this morning. Cloudflare says ~15k unique visitors in the past few weeks FWIW.

qchris• 5 years ago

I literally did the same thing last night! A DO droplet hosting a statically-generated site composed of html generated from some markdown and sandwiched between some html headers, then served up by the Rocket[1] framework.

It's my first website, and I think the total size of the backend is something like 40 LoC. I've never had any kind of analytics (or needed them), but I thought it would be neat to see what was going on. It took me all of 20 minutes to sign up, add the single Plausible <script></script> line to my index files and headers, and there it was. As someone with literally no other web experience, I can't imagine it having been easier. The cost of my website is now a whole $12/mo instead of $6, and I agree that it feels kinda nice to support a service that seems to place value on privacy and openness.

[1] https://github.com/SergioBenitez/Rocket

austhrow743• 5 years ago

Analytics are business intelligence. Showing which of your content works well and which doesn’t.

The idea that there should be no issue sharing them is absurd. Yeah just publish the minutes from your marketing strategy meetings as well while you’re at it. Invite your competitors to go over your books with you and your accountant and then arrange meet and greets with your most profitable customers as dessert.

slimsag7• 5 years ago

Sure, in the same way that code is intellectual property and should never be shared, and all ideas should all be patented in case they become worth something.

In all seriousness, I'm OK with that as long as people are honest about it. Just say "We track our users to collect business intelligence." and I'm happy.

themoose8• 5 years ago

This is a good point and I agree for most businesses, you wouldn't want to publish your bi. But slimsag makes an interesting & valid point: that if you are confident enough in the privacy policy of your analytics, you should be able post it publicly without alteration.

aflag3• 5 years ago

I am not disagreeing with you, but imagine if all business were completely open about all those things. Yeah, maybe your particular business would fail, but the net result would be the truly most effective business to succeed. Which would be an overall win for society. It would also mean that small players would have more of a chance, because they'd be able to build on big players failures/successes more easily. I suppose counter argument would be that it would inhibit people from taking risks, because the gains would be shared, while the failures would not.

rozenmd• 5 years ago

I also use Plausible for OnlineOrNot (https://onlineornot.com) - but there's no way I'd make it public.

Content marketing is part of the secret sauce these days.

literallyWTF• 5 years ago

Whenever I read these hyperbolic statements on Hackernews I always end up rolling my eyes.

Really? It’s spying now?

hervature• 5 years ago

Don’t we call classified (not public) foreign analytics spying?

hobofan3• 5 years ago

Great, now you've made it possible for everyone that links to your site to track people by adding arbitrary UTM parameters which can be uniquely identified. POC: https://opendata.hexops.com/devlog.hexops.com?utm_campaign=p...

Instead of what solutions like Plausible are meant to do (only give a responsible well-defined party access to the data), now everyone can view your website users potentially PII.

slimsag7• 5 years ago

Plausible literally provides a toggle to make it public in their UI: https://plausible.io/docs/visibility

Sure, someone could send out unique links to random `utm_campaign` IDs and see "did the person click it?" and I agree that's a bit nasty.

But even if you do convince someone to click _your_ link, what good does that do? We can see you yourself clicked that link from Germany on a mobile phone, and that's literally it. We can't get your IP address, or any PII - it's not stored.

You were able to verify what level of information my site is collecting *because* it is public. That is simply better than me spying and recording everything you do and you just having to "hope I never accidentally release it or sell it to someone" as is the case with modern tracking.

nicbou• 5 years ago

I switched my professional website to Plausible, and I am very happy with it. It gives me enough data to make business decisions, without tracking my users across websites. I also prefer its interface, which quickly lets me combine filters by clicking on things (visited this URL + triggered this event + came from this source).

It was satisfying to remove the cookie notice, and to update the privacy policy with "we do not collect any personal data". That's much more in line with how I want to operate a website.

On my personal blog, I got rid of analytics altogether. I don't care to know who visits my website.

pjc502• 5 years ago

privacy advocates still go for "I have this hidden treasure trove of analytics about my site that you cannot access"

That's .. private? Why is it surprising that privacy advocates would say that this is private?

aflag3• 5 years ago

I think he's looking at it from the user's perspective. Some privacy advocates may say that analytics have to be private not to make user's private data public. However, the point was that the only reason making the data public would violate user's privacy is that the data is already violating user's privacy, although only one party can access it.

loloquwowndueo• 5 years ago

Before google analytics juggernauted the problem space, there were a bunch of log analysis tools which produced metrics similar to what the article describes, and more, locally and based on nothing but server logs.

A ton of them still exist : things like webalizer, goaccess, and the venerable analog. See the self-hosted category here (https://en.m.wikipedia.org/wiki/List_of_web_analytics_softwa...)

Like the article, I recently realized that, while I did use it, I didn’t take advantage of google analytics at all so decided to nuke it from my sites. I had additionally been running webalizer for over a decade, but recently replaced it with goaccess (took about 10 minutes to set up) which is more actively maintained and have been quite happy with the basic metrics it produces. Sure it won’t stalk a user as they browse my site but I don’t care, just having an idea of how many hits I get, from which countries and which browsers is enough for my few personal and hobby sites.

eli4• 5 years ago

I get a LOT of bot traffic with bogus user agents. Google Analytics does an okayish job dealing with it. Looking at web server logs and trying to reconstruct traffic patterns from just real users would be real chore.

Seirdy2• 5 years ago

I mean, that sounds like more of a feature than a bug. You shouldn't know exactly what people are doing on your site (too invasive), but this approach gives you a rough idea of which requests are being made.

I personally find it to still be too revealing in some cases, which is why tools like the nginx IPscrub module[0] should be the upstream default IMO.

[0] https://github.com/masonicboom/ipscrub

kevin_thibedeau• 5 years ago

I abandoned GA because they wouldn't do anything to fix the problem with referer spam. Unless they've finally fixed it, a lot of the stats they produce are from non-users.

oneeyedpigeon• 5 years ago

Good if you have access to server logs, but not all hosts offer that. GitHub pages, for example.

codingdave• 5 years ago

True, but that feels more like a critique of Github Pages than a valid reason to keep using Google Analytics.

megous• 5 years ago

nothing a "img = document.createElement('img'); img.src = 'https://your-server.net/t=' + Date.now() + '&u=' + encodeURIComponent(location.href); document.body.append(img);" won't fix

ximm• 5 years ago

I was deeply amused by the title. In my world using shell scripts to analyze access logs is the obvious solution and using google analytics is the exotic approach that you might come up with in very few special cases.

mobilemidget• 5 years ago

AWK & SED :) one of the better books I bought when studying. Often parsing a logs to find some things that other logging don't record/show for a manager. Often you hear the same phrase 'You can check logs like that, and you don't need a browser? How... magic....'

kevinyew• 5 years ago

I wish there was something like this for JAMstack/JS only websites that you host on eg Netlify. I know Netlify has their own analytics but they're paid, which is fair enough, but hard to bite the bullet when Google Analytics is free.

nindalf3• 5 years ago

I believe Cloudflare’s analytics are free. I use it because they don’t set cookies and don’t share the collected data with anyone.

hanniabu2• 5 years ago

I'm using netlify by building directly from github. How would I go about inserting cloudflare into this?

shubhamjain• 5 years ago

If you want more out of your access logs, you can also use two free IP Tools we have recently created at IPInfo. This maybe less useful if your aim is to understand the website usage, since we are only creating a report based on IPs. But it can help you understand the type of traffic you're getting.

Summarize IPs takes in a list of IPs to create a report, like this[1], about location of IPs, if they are VPN/Proxy/Tor, and type of traffic. There's a limit of 1000 IPs here, but that can easily be circumvented by creating a free account.

Map IPs plots them on a world map, like this [2].

Both of them are free and can be used from command line. We will also filter out the text to find IPs, so you can just pass raw access logs.

[1] https://ipinfo.io/summarize-ips/result?id=320d5003-a75f-42c3...

[2] https://ipinfo.io/map/5b49e994-ec7a-416c-8140-b311364f6160

29athrowaway2• 5 years ago

In these scripts...

    /usr/bin/awk '{print $7}' /home/logs/access_log.old | /usr/bin/sort | /usr/bin/uniq -c | /usr/bin/sort -r

1) If your PATH is well configured, you can omit the path to the binaries (i.e.: "/usr/bin/sort" becomes just "sort")

2) Instead of using awk, you can just use cut.

    # Delimit this line by the space character
    # and get the 7th field
    cut -d " " -f 7 /home/logs/access_log.old

3) Then, there's this:

    # This is not correct (lexicographic sort)
    sort | uniq -c | sort -r

    # This is correct (sort numerically)
    sort | uniq -c | sort -rn

Why? because otherwise you get 1, 10, 2, 3, 4, 5, 7, 8, 9 instead of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10.

Try it yourself:

   seq 1 10 | sort     # lexicographic sort
   seq 1 10 | sort -n  # numeric sort

4) If you are just interested in the top 10, you can add

    sort | uniq -c | sort -rn | head -n 10

wiredfool• 5 years ago

Awk is friendlier than cut in that it can easily collapse double spaces into one deliminator.

Cut is much more painful to use.

thegeekpirate• 5 years ago

Full paths are probably used to avoid malicious $PATH issues (https://cwe.mitre.org/data/definitions/426.html).

It should just be `sort -rn | uniq`, no point in sorting the list twice.

29athrowaway2• 5 years ago

That's incorrect.

"uniq -c" adds a column with the count of unique elements.

For the purposes of returning the most frequent elements, you can only do the final sort once that column has been added.

dybber• 5 years ago

An alternative to rolling your own scripts is to use GoAccess for web lig analysis, which can both generate HTML statistics or you can open its ncurses based UI: https://goaccess.io/

pugworthy• 5 years ago

This is basically a mid 90's approach to analytics. Glad it works for the author.

je42• 5 years ago

running posthog, which is open source and can be selfhosted ;) stack requires a server instance + redis + postgres for the opensource deployment.

00deadbeef• 5 years ago

Sorry to hijack but I have a slightly related question: anyone know a good privacy-focussed analytics tool for apps that works nicely in React Native? Sort of like plausible.io but for apps

marvinblum• 5 years ago

One of our [0] customers does something like that. He has developed a .NET framework [1] to send clicks to our API and monitor how his apps are used from our dashboard.

[0] https://pirsch.io/

[1] https://github.com/Brouilles/pirsch-dotnet-sdk

feistypharit• 5 years ago

Checkout country: https://count.ly/. can be pretty easily self host, supposed web and apps.

jerrygoyal• 5 years ago

If you don't want to pay for privacy-focused analytics (in case it's some non-commercial project/personal blog) then recently launched cloudflare analytics is the way to go.

FalconSensei2• 5 years ago

That's what I did a couple days ago. Moved my blog from GH pages to cloudflare pages (and DNS). So far so good, and I'm seeing more views than before, when using goatcounter and seems more accurate.

tarkin2• 5 years ago

You can also log requests and filter out obvious bots. Reading the logs, and using basic filters, gives you a decent idea very quickly. I suspect Google Analytics became so popular because people stopped running their own servers. And these days it's ridiculously easy to run your own server.

bgrgndzz2• 5 years ago

Google Analytics has many more use cases than looking at vanity metrics to be replaced by a shell script. Something like https://hockeystack.com gets closer.

coderager32• 5 years ago

is this cookieless?

bogwog2• 5 years ago

The landing page claims that it doesn't, but their own site has this message:

Hey! We use cookies to track you through our website and use this data to optimize your experience! Thanks!

So apparently, their own software isn't good enough to optimize their own site. At least, not without cookies.

tux2• 5 years ago

I've been doing similar but with much more advanced bash script since about 2010 :-) I haven't used Google Analytics, after I learned how they use this data. I just never even thought about sharing this, because I thought no one would care. But I find like minded people on HN all the time ^_^ It's much cooler if you use this as a live script to watch for traffic. You can filter out only the pages you want to see for example. If there is any interest, I can share my scripts ;-)

samstave• 5 years ago

Please share

fidesomnes• 5 years ago

2nd share request

zeeshanqureshi3• 5 years ago

Joining the line. Please do share.

tux2• 5 years ago

Nothing special really, but you asked for it ^_^ https://github.com/tuxys/bash/blob/main/tail.sh

This script is for nginx web server, but I'm sure with some modification it will work on other web servers like apache.

EDIT: Later after some rewrite, I will also publish "stats.sh" a tiny script that generates daily/monthly/yearly static logs.

zeeshanqureshi3• 5 years ago

Thanks!

jeffreygoesto• 5 years ago

https://i.pinimg.com/originals/18/6a/26/186a26ae732311033c67...

tootie• 5 years ago

Half the point of GA is tracking users across sites for ad targeting. It is simply not possible without 3rd party cookies. First-party analytics can always be done with log analyzers or even just a CNAME on a hosted tool. That's just not what most marketers want or need.

ohyash• 5 years ago

Replaced GA with goat counter and called it a day. I'm not disappointed.

FalconSensei2• 5 years ago

I used goatcounter for a while, it's very nice. Def. a great replacement for google analytics if you only want to see if there are visitors.

bernardand• 5 years ago

I’d actually argue that this would have detrimental effects. What’s stopping a company from visiting your site, seeing what’s popular, and stealing it all for themselves?

zeeshanqureshi3• 5 years ago

Lots of paid recomendations on this thread. There's also Umami[1] which is free, open source and self hosted.

I've not used it yet, just have it bookmarked for my next project.

[1] https://umami.is

mikojp• 5 years ago

Yep, sounds like a lot of the authors found a cheap way to market their thing. I've been reading a bunch of the same type of Google 'replacements' promoting the same thing over and over. TBH, it gets annoying over time. HN should do something about this.

bbkane• 5 years ago

This works well if you run a server, but for a static site, you need client-side JS. I recently found 2 free AND open source services (though I did donate): GoatCounter[0] and Counter.dev[1]. I've added them to my static site[2]. Naturally they disagree on my (low in any case) traffic, but they're still fun to look at :)

[0] https://www-bbkane-com.goatcounter.com/

[1] https://counter.dev/dashboard.html?user=bbkane&token=MEzACRA...

[2] https://www.bbkane.com/

_zjfx• 5 years ago

The blog is generated by Hugo and rsynced to NFS. It is a static website.

Replace Google Analytics with a shell script

Related Stories