kiyoto

While I'm happy to hear about a great success story of a great piece of open source software, Elasticsearch has done a great disservice by making application developers lazy about learning the ins and outs of various analytical/transactional/storage backend systems.

Echoing other commenters, Elasticsearch is hardly the best tool for many kinds of analytics. In fact, it is strictly not a good tool for several use cases. For starters:

1. It's not good at joining two or more data sources

2. It's not good at complex analytical processing like window functions (for example to calculating session length based on the deltas of consecutive timestamps partitioned by user_id and ordered by time).

Of course, it's also good at many things like simple filtering and aggregation against "real-time" data. Being in-memory really helps with performance, and with right tools, it's horizontally scalable. Elastic's commercial support is also not to be discounted.

However, as an old OLAP fart who spent years optimizing KDB+ queries, I am deeply concerned about the willful ignorance of data processing systems that I see among Elasticsearch fans. Just take my word for it and study Postgres (with c_store extension) and other real databases, in-memory or otherwise, open-source or proprietary, so that you won't be shooting yourself (or future co-workers) in the foot, trying to shoehorn Elasticsearch and its ilk into suboptimal workloads (To be fair, I see a similar tendency among Splunk zealots).

What surprised me a bit is that pg decided to use the word "bias" without any clarification, considering his background in computer science and AI.

Anyway, I think pg's whole argument is rather moot because the three assumptions that he states are incredibly difficult to measure (Part of the reason why it is very difficult to argue for or against affirmative actions without coming across as "biased").

This reminded me of an interview of David Foster Wallace

"...here’s this fundamental difference that comes up in freshman comp and haunts you all the way through teaching undergrads:　there is a fundamental difference between expressive writing and communicative writing. One of the biggest problems in terms of learning to write, or teaching anybody to write, is getting it in your nerve endings that the reader cannot read your mind. That what you say isn’t interesting simply because you, yourself, say it. Whether that translates to a feeling of obligation to the reader I don’t know, but we’ve all probably sat next to people at dinner or on public transport who are producing communication signals but it’s not communicative expression. It’s expressive expression, right? And actually it’s in conversation that you can feel most vividly how alienating and unpleasant it is to feel as if someone is going through all the motions of communicating with you but in actual fact you don’t even need to be there at all."

"Conversations with David Foster Wallace" (Literary Conversations Series, page 113

A big thing that pg seems to be unaware is that most of us are expressive, not communicative, when we talk. Stylistically, it's true that plain English is the way to go. However, the deeper problem lies in our (in)ability to communicate our thoughts, in writing or in speech.

I agree. One of my mentors recommended that I write down my own version and share it with my team. Here is my attempt: http://kiyototamura.tumblr.com/post/130937953602/good-market...

I actually began using Acme a couple of months ago: it's kind of interesting how much I have accustomed to mouse-driven interaction and lack of syntax highlighting. Humans indeed are creatures of habit.

You are preaching to the choir here. Fluentd (as a proxy for my logging-related beliefs) pushed Docker to have logging long before the logging driver, and now it is one of the officially supported logging drivers.

What you don't seem to realize is that the cost of centralized logging is not always worth it. Machines are ephemeral and so are many application related problems. It's one thing to counter the OP saying that centralized logging has merits (and I believe the OP agrees with that statement) and another to say centralized logging is always a must.

I think you are conflating between the needs of centralizing logs and having them around at all. The OP is saying that always centralizing them might not always make sense, and I tent to agree (and I say this as someone who maintains a popular open source log collector)

If I were to play the devil's advocate, the real needs for raw log data in a centralized location is for folks outside of Ops: data analysts and data scientists.

I think there are two kinds of logging that's conflated into two in the industry: logging for devops and logging for analytics.

For logging for devops, I 100% agree with you. Looking at application metrics rather than raw logs is far more productive, and the raw logs should only be consulted after you have triaged the situation based on the metrics monitored.

However, there is another kind of logging, and that's for data science and analytics. Here, it's hugely helpful to have centralized logging. Hell, it is a must. The last thing you want is to have data scientists with a shaky Linux knowledge to ssh into your prod machines. At the same time, logs are the best source of customer behavior data to inform product insights, etc. By centralizing these logs and making them available on S3 or HDFS or something, you can point them there and have everyone win.

Among Fluentd users, we definitely see both camps. As a matter of fact, one of the reasons that I think people like Fluentd is that because it enables both monitoring and log aggregation within it.

Compared to GCP, perhaps, but AWS's support is pretty atrocious too. The real reason AWS wins is because they know how to sell platforms: as you said, it's about more services, more options, and yes, more _selling_. You can't just build the best components and wait for people to try them. You have to go listen to customers and propose how to build what they need using your platform. This is by far the biggest difference between GCP and AWS: the technical salesperson mindset.

Is this document out of date?

No, in the described case, the message can get lost. This is a really unlikely scenario though. The only real-world case that I know of first-hand is using file buffer and somehow being unable to write to disk, possibly because the disk is full. Something like that can be prevented by a fairly routine set of server monitoring alerts.

We'd love to use a ruby-based solution like this, but the docs say it will lose data whenever the receiving end crashes. Any plans to fix that?

Where does it say this? I don't think this was ever the case for Fluentd.

The way it was described in the docs gave me the impression there is no acknowledgement of network writes - if that's true won't even clean shutdowns lose data sometimes?

This is not true. All writes are acknowledged over TCP, at least between Fluentd and Fluentd.

I'm not here to really defend MongoDB, but the amazing thing about MongoDB is that, despite all the vitriol against it, it continues to be used at companies and projects that are far greater than what many naysayers ever touch: Stripe and Wish.com immediately come to my mind.

Also, I am giving MongoDB the benefit of the doubt per Curt Monash's law of databases:

Rule 1. It takes at least 7 years to build a database

Rule 2. You are not an exception to Rule 1.

Interesting. Do you run hekad on the host machine?

That's an interesting idea: right now, we do not have this. However, it should not be too difficult to implement it either as a filter or output plugin.

This feels like they are trying very hard to avoid calling their syslog a syslog

I _wish_ rsyslog and their friends were really that easy to extend, and I say this as a maintainer of Fluentd. Fluentd came about precisely because syslog family of data collectors fall short in certain ways:

1. tag-based data routing: as you get more and more data sources, it becomes very important to keep track of what goes where. In my view, this is one of the key reasons Fluentd is used at many companies, and why Kafka has become popular on the message queue side (topic-based stream modeling)

2. Extensibility: afaik, it's not all that intuitive for most programmers and sysadmins to extend and add new inputs, outputs, filters, etc. for rsyslog and/or syslog-ng. Admittedly, this is a subjective point, but looking at both Logstash and Fluentd's vast lists of plugins [1][2], I feel justified to make this claim.

3. Configurable transport logic: I've never met anyone who is happy with syslog's buffering and/or failovers. Because we've heard so much about this particular problem, when we were building Fluentd (...4 years ago), we took extra care to make buffering and failover easy to configure and extensible.

Happy to answer more questions =D

[1] https://www.fluentd.org/plugins/all

[2] https://github.com/logstash-plugins

One of the maintainers of Fluentd here. Care to elaborate?

Great to see the support for Fluentd and GELF logging drivers =)

Logging is such a big part of production deployment, and it's good to see a couple of solid options in Docker's core.

He is probably a formalist.

It's actually a she =)

Anyway, I do agree with you (and the author) that mathematics has the potential of being a superb pedagogical vehicle in teaching design thinking.

I've been a big fan of your blog and also tried to recruit you at one time to no response!

On the logging front, there is some options now: syslog, GELF and Fluentd.

- logging/monitoring/alerting

As a maintainer of Fluentd [1], an open source log collector now integrated with Kubernetes [2] and Docker [3], happy to see this out =)

[1] https://www.fluentd.org [2] http://blog.raintown.org/2014/11/logging-kubernetes-pods-usi... [3] http://blog.treasuredata.com/blog/2015/07/07/collecting-dock...

Have a good way of querying, analyzing, processing, and securing all this time series data in a way that can handle literally getting >100TB per hour?

Is this data logging the changes or the states of your sensors? If it's the states, then I am guessing most of this is highly compressible. If it is actually 100TB of changes logged, then that's a pretty difficult problem.

How do you deal with events with varying schema?

KDB is nothing like Hive or redis. In fact, there is no prominent open source project comparable to KDB in terms of architecture or use case.

KDB is a fast column oriented database with an APL-like API. Hive is a SQL translation layer for MapReduce/Tez and Redis is a key-value store with rich data structures.

Open source lets you have full control of your future, while not needing to write all the code yourself.

I disagree, and I say this as one of the maintainers of a decently popular open source project (https://www.fluentd.org)

It's true that open source allows you to tinker and that it lets you jump start your project. But as you point out yourself, a huge part of your software cost is maintenance, buy or build, open source or proprietary. The reason why open source is great is that as long as you can avoid forking the project, you can reduce your maintenance cost by leveraging community contributions and support.

However, the flip side is that you do not have full control: the community often has different needs than your own, and it's not always easy to convince the community so that the open source project caters to your needs. This is double true if you are _just_ a user.

The alternative of course is to create a fork and either take it in-house or try to get the community to rally behind your fork. In either case, you either lose the community's support entirely or reduce its effectiveness, and the same question/burden of long-term maintenance strikes you back.

I agree. My employer Treasure Data (http://www.treasuredata.com/product) has been offering exactly this (SQL on schema-on-read data) for the last three years as a cloud service. But in reality, people "get" schema-less much better than "schema-on-read" thanks to all the marketing that MongoDB folks have done for "schemaless".

I've definitely recommended your service to my friends looking to learn SQL.

Also, I made my share of contribution to the SQL pedagogy by writing blog articles on "SQL for Excel Users":

http://blog.treasuredata.com/blog/2014/12/05/learn-sql-by-ca...

Apparently, Haskell had a side effect.

RIP. What a great mind =/

Unstructured logging, while easy for application developers and ops people, is a major source of headache for data scientists/engineers.

I am a big proponent of JSON-based, semi-structured logging. Most of log data today can be parsed with reasonable rigor at source, and doing this before shipping data to the backend saves so much future agony.

I recently blogged about this here: http://radar.oreilly.com/2015/04/the-log-the-lifeblood-of-yo...

An open source maintainer here (https://www.fluentd.org) We can always use your help for our docs -> https://docs.fluentd.org and https://github.com/fluent/fluentd-docs