A request to cloud providers: Give us access to the logs, please?

http://gudado.com/articles/logs-in-the-cloud

by tremt • 15 years ago

28 14 15 years ago

wccrawford• 15 years ago

"This is not hard to do."

That's a big assumption. I've worked with systems that wouldn't have been able to provide this without a LOT of work and upgrades, and they weren't anywhere near the size if Google and Amazon.

ddbbcc• 15 years ago

For a company at their size, they probably already have those internal logs in place (for debugging, security, etc). Would be nice if they opened it up to the en d users.

andrewjshults• 15 years ago

Debugging/security logs are probably one of the last things that I'd want to hand over to customers without some serious scrubbing first. Either on purpose or accidentally (backtraces, var dumps) I've found things that I wouldn't want turned over to customers (passwords, access tokens, etc.) in logs of various project I've worked on. That var dump on a random object that was put in when tracking down some issue that could only reproduce on production seems innocent enough until someone adds an object to that one and all of a sudden the logs have passwords/account balances/etc. in them. Even if you are scrubbing them, handing over the debug/security logs just seems to introduce too large a point where you could accidentally leak information. The safe solution is to build out logging as a feature and then I completely agree with patio11 that if you need audit trails etc. you're in the enterprise game now and it (rightfully so) comes with a price tag to match.

peterwwillis2• 15 years ago

They have the logs. They don't have it in a form which they can give to you, and they don't have an easy way to expose them to you.

Imagine if every piece of data that involves their cloud service is accessed using a custom piece of code and spread across a multitude of locations all over their network, unsorted. Now they have to build an application which can consolidate a specific customer's logs and provide an interface for them to download it. It's certainly "doable", but it's also probably the very last feature they're thinking about implementing.

mkjones• 15 years ago

Even ignoring other commenters' valid concerns about sensitive data stored in logs, this can be a tough thing to scale.

Say the logs live on disk on a filer somewhere. Need to do a security investigation? OK, dig up the file the logs are in for that particular user and grep around. Lots of disk seeks, but pretty fast.

Scaling that to support production-scale queries is a significant undertaking. (Oh, you can only do 100 seeks per second, need on average 100 seeks per query, and desire to service a low demand of 100 qps? Not gunna happen!)

patio11• 15 years ago

This is totally doable, and if someone asked for it for Appointment Reminder, it would turn a $30 a month account into a $5,000 a month account by itself. If you honestly care about audit trails, welcome to enterprise pricing and hope you enjoy your stay.

mseebach• 15 years ago

I don't agree with the articles assumptions on the immense usefulness of this. Services where it makes sense disseminates the info anyway. Amazon sends me e-mails when I buy something. Other sites have RSS feeds of activity.

Why would I want to run intrusion detection on cloud based services? Isn't that why you put it in the cloud?

peterwwillis2• 15 years ago

You mean, why would you want to know if a particular account or IP is attacking your customers/your business?

jgodin• 15 years ago

It would be nice to some more of this.

For Google Apps (for Business, Education, and ISPs), information on logins (both successful and failed) and logouts is already available to the domain administrator: http://code.google.com/googleapps/domain/audit/docs/1.0/audi...

For personal accounts, you have access to SOME of this data: select "Details" next to the account activity information in the footer of Gmail's web interface.

racketeer• 15 years ago

Yes, we could give you the logs,.. but if we have a multiple tenants in one machine, then the logs are going to show those users activity as well... This is a little trickier than made out to be.. Still doable, but some of the companies/products mentioned, (i.e. Salesforce) have customers on there that wouldn't want you seeing they're activity. You could begin to draw assumptions about some salesforce customer you share a machine with by looking at their log activity. That company would then not be very happy with salseforce...

invertedlambda• 15 years ago

Heroku does this - they have a nice console interface for pulling down the logs from your deployed app. Granted, it is not quite the same as Gmail, but it's always a start.

Maybe it's a business idea: email for geeks. I'd use it. :)

btilly• 15 years ago

You can get at some of that information for gmail. See http://mail.google.com/support/bin/answer.py?ctx=gmail&a... for details.

WestCoastJustin• 15 years ago

Almost 10 years ago I worked for a digital imaging company called ACD Systems and we had a major product launch of ACDSee. We were using Akamai as a CDN and we wanted to know the stats about who was downloading.

Think about this for a second.. Akamai is massive with 10,000+ global servers handling massive amounts of traffic. It might sound simple to fetch one users logs but how do you make this simple for the user. Akamai also had a proprietary log format. To their credit we had a couple conference calls and we worked with their engineers to find a solution. They were a great company.

You don't see logs being offered because it is a major pain in the ass to compile these logs from many servers and reduce them for a specific user.

newobj• 15 years ago

http://docs.amazonwebservices.com/AmazonS3/latest/dev/Server...

A request to cloud providers: Give us access to the logs, please?

Related Stories