We ditched Google Analytics

https://spideroak.com/articles/yeah-we-ditched-google

by felipebueno • 11 years ago

490 262 11 years ago

spideroak.com

nateberkopec• 11 years ago

An open-source, self-hostable solution providing 80% of common Google Analytics functionality seems doable to me.

Is there anything out there in this realm? If not, why not?

davidroetzel• 11 years ago

Have a look at piwik: http://piwik.org/

sarciszewski7• 11 years ago

I don't recommend Piwik. Using MD5 for passwords in 2015 is beyond irresponsible.

https://github.com/piwik/piwik/blob/7f375924db9328f20a0b7cb1...

https://github.com/piwik/piwik/blob/6846145992278b52a2a35a8f...

hendi_• 11 years ago

Yes, there is: www.piwik.org

dataewan• 11 years ago

http://snowplowanalytics.com/ is worth considering if you have larger volumes of traffic

marrone12• 11 years ago

Snowplow is great. Super scalable and they have instructions on how to host everything on your own AWS infrastructure.

pg12• 11 years ago

+ 1 for snowplow. We been using it for more than a year now with high traffic sites and it's working great.

nkuttler• 11 years ago

Why not for low volume sites?

zachh2• 11 years ago

What do you use as a front-end?

kordless3• 11 years ago

Self hosted analytics is still centralizing behavioral data on the users. It's not really any better from a privacy standpoint than GA.

jon-wood• 11 years ago

Its nowhere near as centralized as Google Analytics though - at least if you're self hosting that data is confined to the silo which is your own analytics, rather than Google being able to aggregate that with their behaviour on every other site they visit as well.

gpvos• 11 years ago

Not at all. The entire point is that Google is able to track one person across many, many sites. That is simply not possible if each site had its own self-hosted analytics.

lukeqsee• 11 years ago

Like lots of other companies with high traffic websites, we are a technology company; one with a deep team of software developer expertise. It took us only a few weeks to write our home-brew analytics package.

I'm a little curious why they decided to go this route instead of using one of the open-source solutions. Aren't there good solutions to this problem already?

BinaryIdiot• 11 years ago

I was curious as well and just assumed the usual NIH (not invented here) syndrome. Web analytics was so mature before Google bought Urchin and turned it into Google analytics. Since that time countless open source projects have sprung up (pwik was the first that came to mind). Google for open source alternatives brings up thousands of pages of projects.

Writing your own is easy for the basic stuff. When you want to move beyond the basics, as Spider Oak will find, it becomes much more difficult.

buro98• 11 years ago

Why not move to push GA data server-side?

Trivial to set-up, immune to adblockers affecting the completeness of data, prevents the write of tracking cookies, leaves data and utility of the GA dashboard mostly complete (loss of user client capabilities and some session-based metrics).

This is the route I'm preferring to take (being applied this Christmas via https://pypi.python.org/pypi/pyga ).

One may argue that Google will still be aware of page views, but the argument presented in the article is constructed around the use of the tracking cookie and that would no longer apply.

I'm shifting to server-push to restore completeness, I'm presently estimating that client-side GA represents barely 25% of my page views (according to a quick analysis of server logs for a 24hr period). I'm looking to get the insight of how my site is used rather than capabilities of the client, so this works for what I want.

JupiterMoon5• 11 years ago

People don't care about the cookie or any of the details of the implementation. They care about being tracked across the whole internet. If you are still contributing to that then you are disrespecting your customers. I hope that I am not one of them.

buro98• 11 years ago

Did you read my comment?

You will have no GA cookie from any of my sites, I am not recording client identifying things or capabilities. It is a server-side push of GA and avoids all client-side interactions.

It is merely, "A page has been viewed, this one: /foo/bar?bash".

There's nothing in there that is tracking you. I'm not even embracing the session management aspect.

I get to use the tool that is best-in-class, in a way that lacks capability to track you.

ufmace• 11 years ago

Except that basically nobody cares about "being tracked across the whole internet", as shown by GA, Facebook, etc being on virtually every popular website and nobody noticing or caring at all. If you care enough to make even the most trivial change in behavior, then you're optimistically 1 in 1000.

ytechie• 11 years ago

I agree. Server-side analytics were actually fairly mature before Google came alone. It's just more complicated in some cases, but manageable. The biggest downside these days would be SPA apps since they are not necessarily touching the server in any regular way.

sparkzilla4• 11 years ago

Yet they are still listed on Google search [http://bit.ly/1ICwud0]. I guess it's easy to have principles when it doesn't cost you money.

AdmiralAsshat2• 11 years ago

How exactly would they stop themselves from being listed on Google? A right to be forgotten request? Their decision was to ditch Google Analytics, not to disappear from the web.

sparkzilla4• 11 years ago

Robots.txt. I think it's interesting that you say that removing yourself from Google is the same as "disappearing from the web" implying that the web is Google. It's not. Perhaps they should use alternative forms of outreach, something I am considering with my company.

vive-la-liberte• 11 years ago

robots.txt. Still, I see no reason to remove the site from Google search.

Tinyyy• 11 years ago

How does being listed on a search engine affect their users’ privacy? The users chose to use Google themselves, they could well use something like DuckDuckGo if they wanted to. They’re avoiding 3rd party tracking on their users, not boycotting Google.

baldfat• 11 years ago

This has nothing to do with search. This has 100% to do with site tracking with Google Analytics.

anExcitedBeast• 11 years ago

I don't understand your complaint. 1) They freely admit that they use Google products, and 2) Google search is a way to be discovered by curious users. SpiderOak is looking to avoid giving excessive user data to Google, which is a totally separate issue.

eljimmy• 11 years ago

Is it just me or is this a click-bait title with hollow content?

sparkzilla4• 11 years ago

It is. It's no big deal to stop using Google Analytics. It is, however, a big deal not to use Google Search, something I am considering for my company.

mort96• 11 years ago

Well, the title says they stopped using Google analytics, and the article explained that they stopped using Google analytics, why they did it, and what they're doing instead. You may not find it interesting, but the title clearly reflects the content, so I'm not sure how it's click bait.

trebor• 11 years ago

To any of the SpiderOak team: thank you.

It's more than just the tracking cookie, though. It's also about Google aggregating all its website data into a unified profile. The data they have on everyone is frightening—all because of free services like GA.

tajen• 11 years ago

Yes, thank you SpiderOak, even though I don't use you: High profile companies quitting GA means we get aware of alternative solution. Today, I've learnt about http://piwik.org .

kordless3• 11 years ago

Sadly, we didn’t like the answer to that question. “Yes, by using Google Analytics, we are furthering the erosion of privacy on the web.”

The only thing "wrong" with using an analytics service to better understand your customers is that it places all knowledge of visits, including ones that wished to be private, in a centralized location. This can be useful in providing correlation data across all visitors in aggregate, such as which browser you should make sure your site supports most of the time.

In other words, there exists some data in aggregate that is valuable to all of us, but the cost is a loss of privacy for smaller sets of personal data.

If individuals don't want certain behaviors analyzed by others, then they shouldn't use centralized services which exist outside their realm of control. These individuals would be better off using a "website" that is hosted by themselves, inside their own four walls, running on their own equipment. A simple way for SpiderOak to address this is to put their website on IPFS or something similar.

I appreciate the fact that SpiderOak is thinking about these things. It's important!

gvb• 11 years ago

The irony for me is that I am mostly invisible to Google Analytics, and thus the companies that rely on GA, because I mostly browse with JavaScript disabled. (When I need JavaScript, I usually crank up an incognito instance of Chrome and close it immediately when done, so I'm mostly anonymous to GA even then.)

When they go "old fashioned" and datamine their web server logs, they uncloak me. :-/

rogeryu• 11 years ago

I'm doing my part. I'm moving to DuckDuckGo for searching more and more. It's a process. Google does have better results. For work I still rely on Google, for private stuff I use https://duckduckgo.com/

And for the sake of ducks, I'm eating less meat as well. No more chicken - too much antibiotics, and as little meat as possible, only when it's worth it, so great taste and good quality.

Xavez• 11 years ago

You can also try https://search.disconnect.me.

Albright4• 11 years ago

I'm a big DDG fan too. I don't really notice their results being "worse" than Google's (but maybe that's just because I haven't used Google for so long). The Bang feature is also very handy once you get in the habit of remembering to use it. https://duckduckgo.com/bang

theandrewbailey2• 11 years ago

More and more, if I'm not satisfied with DDG, I will Google that term, only to get the same stuff.

gerty2• 11 years ago

Do you also experience slower response times at DDG?

Here in Europe 'ping -c 5' gives an average of about 10ms for google.com and 30ms for duckduckgo.com. Since search is such a fundamental part of browsing, this is very noticeable.

_lce0• 11 years ago

Kudos for this!!

it's interesting that still there's meta, probably leftover

    <meta name="google-site-verification" content="pPH9-SNGQ9Ne6q-h4StA3twBSknzvtP9kfEB88Qwl0w">

EDIT: wow, thanks for your answers guys!! so nice to see Cunningham's law in action ;)

veeti• 11 years ago

That is for webmaster tools, not analytics.

chainsaw10• 11 years ago

IIRC that meta tag is also used for verification for Google Webmaster Tools

artursapek• 11 years ago

That's for Google's webmaster tools, which is another service Google offers (although in this case, you get to peer into their data)

stonedge• 11 years ago

This could just be for Google Webmasters which gives them information on their search presence.

stanleydrew• 11 years ago

Surprised that they don't use the DNS verification option. Bloating your HTML for every request is unnecessary.

cm21872• 11 years ago

The other thing is that google analytics is on many adblockers lists, precisely for that reason. As adblockers are getting widespread, the analytics is going blind.

Albright4• 11 years ago

I've been running a blocker to block GA and other junk on my PC, but I imagine I'm in a statistically insignificant minority. And I still can't block them on my iPhone unless I disable JavaScript entirely (though I'm running iOS 9, I'm not able to install a blocker for some reason; I guess Apple arbitrarily doesn't support them on my older iPhone model).

djrogers• 11 years ago

I guess Apple arbitrarily doesn't support them on my older iPhone

It's not arbitrary - it requires a 64 bit CPU (of which Apple has now shipped 3 generations of).

Kristine1975• 11 years ago

If your device is jailbroken (not sure if there's a jailbreak for iOS 9), you could add entries for GA to its hosts file. I use these on my desktop PC:

  127.0.0.1 www.google-analytics.com
  127.0.0.1 google-analytics.com
  127.0.0.1 ssl.google-analytics.com

AFAIK ad blockers are only supported on iOS devices with 64 bit CPUs.

cm21872• 11 years ago

I could install an adblocker on iOS9, and you can customize the block list.

I don't think you are a minority. I understand adblocking usage is around 20%-ish now.

What I don't understand is people who use adblockers but still login to their google account on chrome. It sorts of defeat the purpose...

vlunkr• 11 years ago

If anyone is looking for a good blocker for stuff like this, I recommend ghostery. I set it to block everything by default, and whitelist the few things I want. It doesn't block scripts served by the site you are on, so it doesn't totally break your browsing experience, like others do.

eponeponepon• 11 years ago

    It took us only a few weeks to write our home-brew 
    analytics package. Nothing super fancy yet now we have 
    an internal dashboard that shows the entire company much
    of what we used analytics for anyway - and with some 
    nice integration with some of our other systems too.

I never quite grasp how the above isn't just a matter of intuition to anyone working in the tech sector. Google Analytics thrives on developers' laziness in my opinion.

And to echo other posters: SpiderOak deserve thanks. If I find myself with any need for a service like theirs, I know I'll be looking at them.

rplnt• 11 years ago

Aren't there self-hosted analytics anyway? Piwik[1] comes to mind first, but I'm sure there are many.

1. https://piwik.org/

Albright4• 11 years ago

Unsurprisingly, Wikipedia has a list: https://en.wikipedia.org/wiki/List_of_web_analytics_software

billmalarky4• 11 years ago

Piwik is incredible. But it should be noted that it does provide a scaling challenge for high traffic use cases (> hundred million actions per month), and hosting your own analytics is expensive.

I bring this up because people had been slamming moot for using GA on 4chan instead of piwik without understanding why.

vermontdevil• 11 years ago

Does the adblock/ublock etc block this as well?

Am looking to use this in lieu of Google Analytics.

sarciszewski7• 11 years ago

See https://news.ycombinator.com/item?id=10697045

Piwik is still using (unsalted) MD5 for passwords in 2015, and probably will still be using unsalted MD5 in 2016.

beerbajay• 11 years ago

Google Analytics thrives on developers' laziness in my opinion.

It's almost never "developers" who are deciding to use GA; it's middle managers or marketing departments.

k__2• 11 years ago

Well, it's not a thing to implement in a few days, but a few weeks.

That maybe no option...

fenomas• 11 years ago

Isn't GA's main draw its close integration with adwords and whatnot? The dashboard and UI seem pretty clearly aimed at someone who needs to manage their spending on google marketing services, not on someone who needs to count pageviews.

So it's not hard to imagine marketing wanting it; presumably it provides them a lot of value that wouldn't be easy to recreate in-house.

jkestner• 11 years ago

And GA is inscrutable. I don't use it very much because it's got way too many layers of abstraction. It was fine before as Urchin. Maybe this is a category like email clients — there should be a sustainable paid product that doesn't suck.

morninj• 11 years ago

Maybe http://haveamint.com/ is what you're looking for? (I'm not affiliated--just a former user.)

fiatjaf2• 11 years ago

There's also http://get.gaug.es/, which seems great.

I have gone through their trial, but now I think I will register for the Solo account ($6/mo).

efuquen• 11 years ago

I don't think it's a matter of laziness. More so where is it best to spend your expensive/valuable developer resources, on the product or some home-baked analytic's framework?

I applaud SpiderOak, but they are much different from most other sites. They have privacy conscious customers to begin with, this is something that is good press for them and probably a net positive on their bottom line for doing it, not the case with most other sites. Also it's something they are doing after having a very mature product for many years, clearly not the first or most important thing they needed to tackle as a company.

IE6• 11 years ago

Agreed - for some cases just pasting the GA snippet onto a site is sufficient. For others you should add events and such. For others you must roll your own.

marcosdumay• 11 years ago

If it's worth A/B testing your site, it's worth doing it with a tool that understands your costs and revenue structure.

GA is mostly used by people that don't need it, yet want to pretend they get actionable data out of it.

digi_owl• 11 years ago

thrives on developers' laziness in my opinion.

Frankly most of what i read out of the tech world these days seems to be about pandering to developer laziness.

All manner of APIs and services seem to exist in their current form simply to extract rent from developers that don't want to do back end "dirty work".

jessaustin2• 11 years ago

Being paid for doing work by those who need the work done is the opposite of "extracting rent".

andybak• 11 years ago

Laziness is a fine quality to have in a developer: http://threevirtues.com/

ascendantlogic• 11 years ago

That starts going down the path of the "not invented here" mindset. You could then attribute not hand-rolling every bit of infrastructure yourself as "laziness". Yes, I am lazy to the point that I don't want to hand-roll an industrial-strength RDBMS myself, or the operating system, or the networking protocol, or the key/value store, etc etc.

untog• 11 years ago

And I never quite grasp why many people working in the tech sector are insistent on reinventing things that already exist. Such thinking thrives on developers' personal sense of exceptionalism in my opinion.

seanwilson• 11 years ago

Yeah, a nontrivial app is comprised of so many parts, if you tried to reinvent a few of them yourself you'd never get anywhere. Also, try looking at the commit history and issue lists of seemingly trivial libraries. It's incredibly easy to underestimate how complex something that looks simple at first can be.

redthrowaway• 11 years ago

It's not laziness, it's opportunity cost. For SpiderOak, it makes sense to spend a few weeks of a few developers' time to roll their own analytics. For me, it doesn't. Our customers aren't privacy-focussed. In fact, our app depends on them explicitly sharing [quite a lot of very personal] data with us. I would rather spend that time building something that delivers value to them and us than indulging my personal beliefs about privacy.

raverbashing• 11 years ago

Most developers don't develop (major) libraries, languages and OSs in house, it doesn't mean they are lazy, it means the company need to focus limited resources on their core business.

splatcollision• 11 years ago

Looking for a good npm / express middleware module that does this. Combines privacy concerns + developer laziness!

justinkramp2• 11 years ago

Maybe https://crypton.io/

onion2k2• 11 years ago

If all you want to know is who accessed a site, with which browser, how long for, and which pages they looked at then you could get all that from your webserver's log files without writing any code. On the other hand, to build something that's robust, relatively scalable, works across browsers and devices, and can give you an event watching platform like GAnalytics gives you (eg the useful bit), that is far from trivial.

bluedino2• 11 years ago

> Google Analytics thrives on developers' laziness in my opinion.

Every service does. Pingdom, GA, Olark, Github...

It took them a few weeks to write their own analytics. What features did they not implement? How many people worked on it?

Does your 1 or 2 person startup have 4 weeks to write their own analytics package or do you have more important stuff to do? (I'm betting you do. Like launching your product instead of re-inventing the wheel with analytics)

Retra• 11 years ago

Everything developers don't do is a matter of laziness if you ignore the fact that they might have other priorities.

morgante• 11 years ago

Having implemented two different custom analytics dashboards, it's a lot more complex than you think.

Sure, the basics are easy. But marketers and business people want to drill into a lot of data which is non-trivial to gather.

Unless you have a compelling business case (which SpiderOak does), it's not worth it.

fiatjaf2• 11 years ago

I've replaced Google Analytics in all my projects with my CouchDB-only web analytics service, Microanalytics[1], which I could access from a CLI[2] and worked very well.

But then I started to fall short on disk space for storing too many events. This is a problem.

[1] https://github.com/fiatjaf/microanalytics

[2] https://github.com/fiatjaf/microanalytics-cli

chimeracoder• 11 years ago

I never quite grasp how the above isn't just a matter of intuition to anyone working in the tech sector. Google Analytics thrives on developers' laziness in my opinion.

Unless I'm mistaken, one big difference is that not using Google Analytics means you don't know which Google search pages people used to access your website. That can be a really important difference for some websites.

cgag• 11 years ago

I think you can get that info with webmaster tools without using analytics.

aargh_aargh• 11 years ago

Can't you find out from the Referer header anymore? It's been years since I tried, so it may have changed.

gedrap3• 11 years ago

>> I never quite grasp how the above isn't just a matter of intuition to anyone working in the tech sector. Google Analytics thrives on developers' laziness in my opinion.

Ah, the "not invented here" syndrome!

There are tons of things that you could do "in a couple of weeks" that more or less work. However, it doesn't mean you have to or even that it would be a good idea.

If all developers adopted the attitude that you have expressed, there would be thousands of sad sad developers who need to maintain shitty in-house analytics system because someone once said "I could do it in a week". There are tons of awful CMSes already because someone once said "I could do better than wordpress" / "I could create a better framework" / etc.

In a lot of the cases, GA is just good enough. Sure, you might need to spend some time to explore its features (custom dimensions, etc), there's more to GA than a number of pageviews for a given day. There are cases when GA is not enough. Fair enough. But it's definitely not the majority of the cases.

Sure, it makes sense for SpiderOak given it's target audience. However, there's no need to make such a generic statement about 'anyone working in the tech'.

EGreg• 11 years ago

The answer is open source. Where is the wordpress to google's analytics product?

yoo1I• 11 years ago

A lot of people are replying to the suggestion of implementing your own analytics by calling out it's NIHness.

I've recently been faced with this problem, and a solution doesn't have to be too complex.

There are roughly two parts to an analytics solution: event logging and, well, the actual analytics.

Writing your own logger in javascript is super simple, you're just sending off json objects to be inserted into a elasticsearch cluster. Since you have to define that logging anyhow, the only extra work you need to do is the layer to do the actual ajaxrequests.

What's left is running and defining your queries in elasticsearch.

BAM! Analytics

I realize it's not fit to be used for every situation, but it can so some pretty complex things this way without the hugest amount of effort ...

gedrap3• 11 years ago

I get what you are trying to say and I was one of the NIH-sayers, it totally makes sense in some cases and looks like it made sense in your case. Great! :)

I don't think anyone was saying that GA is always better, it's just more often than not it is. It takes some skill and quite a bit of experience to draw the line at a reasonable place and correctly recognize the trade offs.

ubersync• 11 years ago

Don't the ad blockers disable Google Analytics by default? If I am not wrong, I think uBlock Origin does.

So, I think, as more and more people will start using ad blockers, site owners will start getting less and less accurate stats from Google Analytics, forcing them to implement their own solutions. Hopefully, open source solutions will start providing the best features that Google does.

Moru2• 11 years ago

Anything that is widely used (open source or not) will be blocked because of common names or other patterns that can be recognized and blocked. If you need exact statistics you need to roll your own sooner or later. Or at least heavily customize some other product.

tootie• 11 years ago

    much of what we used analytics for anyway

Until your requirements grow and your stuck building something that was in GA 5 years ago.

oneJob• 11 years ago

How about open-sourcing your product before worrying about improving other products? SpiderOak has been "investigating a number of licensing options, and do expect to make the SpiderOak client code open source in the not-distant future" for a very, very long time now. It's no trivial thing to have a closed source client for a "zero knowledge" service.

https://spideroak.com/faq/why-isnt-spideroak-open-source-yet...

EDIT: I'd welcome discussion, in addition to your up/down votes

gerty2• 11 years ago

I am also concerned with that. That message has been there unchanged for some time now. To be fair, there's a lot of stuff on the Github page, including the Android client under Apache license. Although as far as I can tell, desktop client is not there yet.

prajjwal• 11 years ago

I came here for this exact thing. They said they were going to go open source in 2014 IIRC, and failed to deliver. I have stopped using SpiderOak - how am I supposed to trust them with my most private files when I can't verify that they're not doing anything shady on my machine?

The opening line of this post is amusing. They ought to give thought to fixing their core product first.

Veratyr5• 11 years ago

Not strictly on topic so I apologise if this is unwanted but I thought I'd share my experience with SpiderOak in case anyone here was thinking of purchasing one of their plans.

In February SpiderOak dropped its pricing to $12/month for 1TB of data. Having several hundred gigabytes of photos to backup I took advantage and bought a year long subscription ($129). I had access to a symmetric gigabit fibre connection so I connected, set up the SpiderOak client and started uploading.

However I noticed something odd. According to my Mac's activity monitor, SpiderOak was only uploading in short bursts [0] of ~2MB/s. I did some test uploads to other services (Google Drive, Amazon) to verify that things were fine with my connection (they were) and then contacted support (Feb 10).

What followed was nearly __6 months__ of "support", first claiming that it might be a server side issue and moving me "to a new host" (Feb 17) then when that didn't resolve my issue, they ignored me for a couple of months then handed me over to an engineer (Apr 28) who told me:

"we may have your uploads running at the maximum speed we can offer you at the moment. Additional changes to storage network configuration will not improve the situation much. There is an overhead limitation when the client encrypts, deduplicates, and compresses the files you are uploading"

At this point I ran a basic test (cat /dev/urandom | gzip -c | openssl enc -aes-256-cbc -pass pass:spideroak | pv | shasum -a 256 > /dev/zero) that showed my laptop was easily capable of hashing and encrypting the data much faster than SpiderOak was handling it (Apr 30) after which I was simply ignored for a full month until I opened another ticket asking for a refund (Jul 9).

I really love the idea of secure, private storage but SpiderOak's client is barely functional and their customer support is rather bad.

[0] http://i.imgur.com/XEvhIop.png

Someone12343• 11 years ago

Many of these types of services seem to intentionally cap upload speeds to reduce their potential storage liability (since they're likely over-selling storage to be able to offer 1 TB for $12 with the level of redundancy, staffing costs, etc, needed).

I wonder if that is happening in this specific case? Although if it were the case the vendor should still be honest about it. Just saying they limit uploads to 2 Mbps is better than giving the run-around.

scurvy• 11 years ago

It's definitely possible to offer that on a monthly basis if you model that each customer stays for 36-39 months. Also, I doubt that they are using replicated storage, but are using erasure coding instead. Also, they dedupe before upload, so more cost savings there.

Joona• 11 years ago

Definitely looks like it to me. Took me a good month to back up my (video) files with CrashPlan, as it was using some 10% of my upload.

I think it would be a good selling point for a service like this to allow higher upload speeds.

jewel• 11 years ago

Pricing is falling rapidly for storage. Consider that S3 - IA is $15/mo for a TB, and backblaze B2 can offer 1 TB for $5/mo. I would assume both are making some profit at those price points, so $12/TB/mo should be workable if the service is doing their own hardware.

Backup services especially have low operational requirements for their hardware and network connection, since once the files are uploaded they only need to be verified periodically.

jcfrei• 11 years ago

Yup, I've noticed the same with Wuala. The uploads were pretty slow. I've heard similar complaints from people using OneDrive. I would be very willing to switch to a smaller competitor even if it meant paying more than I do at Dropbox. But from my experience Dropbox is the only provider capable of synchronizing large amounts of data 24/7.

toomuchtodo7• 11 years ago

reduce their potential storage liability

Its to reduce their maximum bandwidth capacity required. I don't see it as a problem, considering their price points. They're selling you storage, not "slam 1TB of your data into our storage system in a day". If you're looking for that, ship a hard drive to Iron Mountain.

EDIT: Even AWS limits how fast you can upload to S3, and built an appliance for you to rent and ship back and forth if you need to move data faster. That station wagon full of tape is still alive and well.

lakeeffect• 11 years ago

The thing that is silly about that is related to cost of acquisition and retention of customers. If a company is able to get more data quicker they are more valuable to the customer and will most like be used and retained by the customer. If organizations are offering storage as a solution while at the same time trying to minimize the costs of that solution, by minimizing the utilization of that storage; they are exchanging fixed costs associated with storage (that should be easily built into pricing) for large variable costs related to customer acquisition, retention, and branding.

newscracker2• 11 years ago

Many of these types of services seem to intentionally cap upload speeds to reduce their potential storage liability (since they're likely over-selling storage to be able to offer 1 TB for $12 with the level of redundancy, staffing costs, etc, needed).

SpiderOak is definitely overselling the 1TB as well as another one that pops up once in a while called as the "unlimited" plan for $149 a year. This is clear from the disproportional pricing structure - $79 a year for 30GB that jumps to $129 a year for 1TB and then to $279 a year for 5TB - which entices users to go for the higher amounts because they appear to be great deals. What people with residential broadband connections may not realize is that a) uploading even 1TB of data will take a long time and b) SpiderOak cannot, and does not, provide any minimum guarantees on the upload or download speeds (assuming everything else in between SpiderOak and the user looks fine).

theandrewbailey2• 11 years ago

I was going to post a comment about how cloud storage is more of a means to move data around rather than back it up, until I dug a little deeper and saw that SpiderOak actually pitches itself primarily as a backup provider. I agree, it needs to be much faster than that.

bluedino2• 11 years ago

> my laptop was easily capable of hashing and encrypting the data much faster than the network was capable of handling it

You are assuming that you are the only one using that uplink and that server

Veratyr5• 11 years ago

Updated my comment:

easily capable of hashing and encrypting the data much faster than SpiderOak was handling it

I can believe that there was upstream congestion somewhere outside my network (speeds to Google, Amazon indicated that there were no issues inside) or that their server was overloaded but the engineer who investigated seemed to attribute it to the client:

Additional changes to storage network configuration will not improve the situation much. There is an overhead limitation when the client encrypts, deduplicates, and compresses the files you are uploading"

jasonsync• 11 years ago

Plug: https://www.sync.com

Veratyr5• 11 years ago

Love the pricing and features but Win+Mac only and no API largely kills it for me as I need Linux access at the very least.

newscracker2• 11 years ago

The only issue I have, which is similar to what I see with some other providers, is that the first non-free plan is a huge jump in storage space and price. If I want a Dropbox replacement, I'd be looking at a 25GB or 50GB plan (just comparing what I have with all kinds of free storage bonuses accumulated over years). Having some more "in-between" plans that are more linear in storage and price would've been an incentive to try this out since I'm not willing to fork $49 a year for 500GB while knowing that my Dropbox usage is less than one-tenth of that.

kbenson• 11 years ago

Is it possible that they are working on batches, and not doing any hashing/compression in parallel with the uploading? It seems feasible from your screenshot that they are getting ~10GB of data at a time, compressing(?) and hashing, and then uploading, and then starting on the next ~10GB.

draw_down• 11 years ago

This comment is ridiculous, and so is the fact that it's at the top. This is supposed to be about Google Analytics, come on.

y4mi• 11 years ago

well, the post is from SpiderOak, so its understandable.

but its an ad hominem argument, thats for sure.

cpach2• 11 years ago

It is off-topic, yes. For me personally it was very valuable however since I’m in the market for a backup application, and I will definitely take Veratyr’s comment into consideration when choosing between the available offerings.

mark_l_watson2• 11 years ago

That doesn't sound good. On the other hand, I use SpiderOak with not a lot of cloud storage use, with clients on OS X, Linux, and until this morning Windows 10. The only problem I ever had was more or less my fault - trying to register a new laptop with a previously named setup.

BTW, why store photos and videos on encrypted storage? For that I use Office 365's OneDrive: everyone in my family gets a terabyte for $99/year and I really like the web versions of Office 365 because when I am on Linux and someone sends me an EXCEL or WORD file, no problem, and I don't use up local disk space (with SSD drives, something to consider).

Veratyr5• 11 years ago

I prefer to store photos and videos on encrypted storage because I want to control who sees them. Storing them on unencrypted storage means I don't have that control, the storage provider does and is kind enough to let me make suggestions.

As for OneDrive, I tried it for a while but it didn't work out. Their clients and web interface were terrible and their API was severely lacking. I expect more functionality when I'm sacrificing my privacy.

I ended up going with Google Drive in the end, as you can get 1TB for $9/month with an Apps for Work Unlimited account (I actually seem to have Unlimited under that plan, which isn't supposed to happen until 4 users). That of course means sacrificing encryption but I trust Google enough to make the privacy tradeoff in exchange for extra features (OCR, Google Photos etc.).

ldehaan• 11 years ago

This has been my experience as well, not to mention how much the client slowed down my machine. It's been really slow going but the client is getting better. I never tried doing the encryption on my side, though, they also do diffs on each file you upload so I imagine that has something to do with the lag. I still use spideroak, they're the only company I'm aware of that encrypts locally and also has done a lot to progress personal security for all of us. So I've gotten used to the slow speeds and buggy software, it keeps getting better so that's a big plus :)

cpach2• 11 years ago

There are other backup applications that encrypt locally before sending to server. Two examples are https://www.tarsnap.com/ and https://www.haystacksoftware.com/

Paul-ish3• 11 years ago

Could the issue be caused by bad peering between your ISPs?

Veratyr5• 11 years ago

If that was the case I'd expect the upload to be consistent but slow. Since it was intermittent, I believe it's an app issue.

pmoriarty• 11 years ago

Now if they could also just ditch Facebook, Microsoft, and Apple, they'd be getting somewhere.

c0achmcguirk• 11 years ago

Spideroak user here. I stopped using Dropbox and started using Spideroak about a 18 months ago. I really like the product. It's not as good as Dropbox in some ways (like automatically syncing photos from my phone) but it really is easy to use. I still have a mobile client on Android and I can keep my files in sync across multiple computers. I pay for the larger storage size and I'm not even close to using it all.

It syncs fast too. Just thought I'd share my experience with people.

gramakri2• 11 years ago

At Cloudron, our vision is to allow companies to host their own apps easily. We dogfood and don't use Google. We don't use analytics on our website (a conscious decision). Our emails are based on IMAP servers and we use thunderbird. We selfhost everything other than email (which is on gandi).

We just entered private beta yesterday - https://cloudron.io/blog/2015-12-07-private-beta.html

dschep• 11 years ago

How's this compare to Sandstorm[0]? It seems like a Closed/SaaS equivalent.

[0] https://sandstorm.io/

gramakri2• 11 years ago

Cloudron and sandstorm are similar projects. I think the main difference is the user experience (also how we handle domains, how apps are packaged etc). You can see a demo of the cloudron here - https://my-demo.cloudron.me/ (username: cloudron password: cloudron). All apps use same credentials (because of single sign on)

qihqi• 11 years ago

Random fact: GA cookie is distinct from adwords(google.com) cookie, and it is illegal for Google to join those (not sure if it is even technically feasible).

sghiassy• 11 years ago

I don't get it. SpiderOak states that they dropped GA because it furthers "the erosion of privacy on the web.”, but then they just started tracking in house.

How is tracking in house more private than GA? The user is still being tracked.

softyeti• 11 years ago

I believe their point was that they want to track their traffic, but when they use a third party like Google, Google provides tracking services for SpiderOak, Google also tracks you as well, which SpiderOak has no control over.

stonemetal• 11 years ago

With it in house it is under their control: they can anonymize it, not collect certain information, they can't cross index it with your traffic from other sites, etc.

distances• 11 years ago

Google likely (or at least theoretically) has the capability to track the user across all the sites that use GA. In-house tracking doesn't allow this.

api• 11 years ago

Instead of rolling your own look at Piwik. It works very well and is basically a GA clone. I actually like it better than GA in some ways. It's easy to set up and you can run it on your own site so you're not contributing to a global tracking fabric.

rwbt• 11 years ago

I've had good experience with Statcounter (http://www.statcounter.com)

rbinv5• 11 years ago

It took us only a few weeks to write our home-brew analytics package.

Unfortunately, there's no way to replicate what Google Analytics currently offers (for free!) within a couple of weeks (or even months). Not with big data sets. Yes, GA does enforce sampling if you don't pay for GA Premium, but the free edition is still one of hell of a deal (if you don't care about privacy).

If you only use Google Analytics as a hit counter, sure, you can do that yourself within a couple of minutes. The advanced features are way more complicated, though (think segmentation and custom reports).

This also begs the question: why not use Piwik?

ssharp• 11 years ago

I suspect most of the people saying "you don't need Google Analytics! Do it yourself!" have never used GA for anything that meaningful. As you begin to really familiarize yourself with your website traffic and understand how to look at your clickstream data in a more investigative an analytical way, you'll start to see how nice GA is and how easy it is to answer your questions.

You also underestimate how ubiquitous GA is because it's free and extremely popular. I'd consider myself an intermediate to advanced user of GA, but for people less experienced, I can easily share stuff with them for complicated tasks or they know how to do a lot of the basics themselves.

In hiring digital marketing people, GA is pretty much on par with Word in terms of familiarity. It's something a lot of people have a basic competence with.

rbinv5• 11 years ago

I agree completely.

GA has become very, very capable in the last five years or so. Combined with their current APIs, you can do pretty much anything you want.

zachh2• 11 years ago

Is there any analytics solution that allows for on-prem data storage? Only one I know of is Adobe Analytics (formerly Omniture).

mattab3• 11 years ago

Yes, check out Piwik.org / Piwik.pro

electriclove• 11 years ago

Why not just use Piwik?

justinkramp2• 11 years ago

It should also be noted that SpiderOak has opensourced many components of their product stack, including Crypton, which is the encryption framework underpinning many of their clients.

The source is at https://github.com/SpiderOak

cpncrunch4• 11 years ago

why does Google and their advertisers need to know about it I would ask

Google is pretty clear about this. The only reason they track you is for advertising, and there isn't any evidence of them using the info for anything else. In fact there is a lot of evidence pointing the other way, such as their insistence on encryption data flowing between their datacenters.

This is Google we are talking about, not Kazakhstan, China or Russia.

Paul-ish3• 11 years ago

Google could eventually use this information to determine your eligibility for a home loan. They have already dipped their toes in this area [1]. With all this data, we have to ensure that it is used fairly (or not at all). There is enough concern about digital redlining that a 2014 report to the white house reports on this [2]. As we know machine learning is quite capable inferring sensitive attributes [3].

This inference doesn't even need to be intentional, machine learning is capable of accidentally picking up on latent variables. Even if your neighborhood (the original redlining) isn't a feature in the original variable, it could be inferred from the other variables.

TL;DR: Your surfing behavior could be used to deny you a home loan one day.

[1] http://techcrunch.com/2015/11/23/google-launches-mortgage-sh...

[2] https://www.whitehouse.gov/sites/default/files/docs/big_data...

[3] http://www.pnas.org/content/110/15/5802.abstract

cpncrunch4• 11 years ago

Lots of speculation about what they might do. You could also say that the US government could use all of your data to spy on people who criticise the government, so they shouldn't have any of that data either.

adrianlmm3• 11 years ago

Still, Google is a corporation whos only porpose is proffit, trust them blindly is foolish.

cpncrunch4• 11 years ago

Their other purpose is "dont be evil". There may be some debate about that at times, but they certainly aren't going to screw their customers. They know that their customer base would evaporate pretty quickly if they tried to screw them.

iamleppert• 11 years ago

I'd hate to be the intern or the guy in charge of keeping that thing running.

TheAndruu• 11 years ago

Make your custom analytics library into a separate product in its own right and sell it or open-source it!

patrickfl• 11 years ago

For anyone here looking for a really good, free,self-hosted, hackable, open source alternative to Google Analytics that's been around for a long time, please consider Piwik.org.

I've been using it for prob 8-10 years and it has never missed a beat. I use it on all my personal / business sites as well as some client websites that are super high traffic.

binaryapparatus• 11 years ago

Analytics, fonts, css. We include it everywhere by default. Then I realized hey we are all giving away too much. My sites now happily run self-hosted piwik, for the last six or so months.

I won't be surprised if in the coming years we hear much more about using google fonts being base to count site access, if there is no analytics in place.

Something1234• 11 years ago

What about awstats or goaccess? Both are great log analyzers, although I like goacess better.

http://goaccess.io/

http://www.awstats.org/

X-Istence2• 11 years ago

The look and feel of awstats hasn't changed since I last used it back in 2004...

mattab3• 11 years ago

Piwik is the new AWStats, check out http://piwik.org/log-analytics/

kevin_thibedeau• 11 years ago

I haven't checked my GA in months since it became clear that Google won't bother doing anything to fix the referer spam problem that makes the stats useless if you don't have a high-volume site. It's not like these abusers are hard to track down but I'll be damned if I'm going to manually add filters to get rid of them every time they come in from a new domain.

emilburzo• 11 years ago

Admin -> select account -> select property -> View Settings (on the view you want) there's a checkbox:

Exclude all hits from known bots and spiders

jliptzin• 11 years ago

Usually I start with Google Analytics but continue to add to our own in-house analytics solution targeting the specific metrics we're interested in tracking. GA often doesn't provide us with the real insights we're looking for, but it's good for the vanity stats.

ksec• 11 years ago

To me, it is the cost that matters. Most other Analytics cost $30 - $50 / 1 Million Pageview / Datapoint. To me this expensive. Even when you scale to 100M it will still cost ~$20/Million.

Piwik doesn't scale. At least it doesn't scale unless you spend lots of resources to tinker with it. Its Cloud Edition is even more expensive then GoSquared which i consider to be a much better product.

What we basically need is a simple, effective, and cheap enough alternative to GA. And so far there are simply none.

We ditched Google Analytics

Related Stories