I don't think Elasticsearch is a good logging system

https://blog.sinkingpoint.com/posts/elasticsearch-logging/

by vimda • 5 years ago

224 111 5 years ago

blog.sinkingpoint.com

airocker2• 5 years ago

You cannot do much analytics on Loki, there is no index so each query is costly

buro93• 5 years ago

You can do a surprisingly lot with Loki.

To get the performance run more queriers (horizontally scaled read path that is in front of the object storage).

We're improving performance constantly, and we already run Loki at a very large scale (multi-region, multi-tenant, etc) with aggressive internal SLOs. We see customers doing network analytics, log analytics, line-of-business data analytics all on Loki and it works really well. Customers also use it as part of Cloud hosted ETL processes and to drive alerts. What we run in the Cloud today is the OSS version on k8s.

That said... if we're missing some use-case or just a scenario that you feel should work and it doesn't work for you, then we really want to learn about it. If there's an out of the box experience that isn't great, then let us know. Feel free to ping me any details you're not willing to put on HN at david.kitchen@grafana.com but I'm very cool with public conversations too.

jarpineh2• 5 years ago

Hi.

I have a question. I have put some time trying to learn the Grafana stack. Heaviness of elastic both self hosted and cloud version has led me to seek for alternatives. Is there a way to get logs into Loki that can be run offline, when jobs have finished and hosts have been shutdown? It seems Loki recommends Promtail agent which looks a bit heavy-handed (https://grafana.com/docs/loki/latest/getting-started/get-log... )

I have relaxed requirements on gathering logs for analytics. I’d much rather just run a tool that goes through log files and ingests them into Loki. Results of several jobs would be compared as a whole.

From what I’ve seen Loki with Grafana can do a lot. For example this video: https://m.youtube.com/watch?v=7zmRhHd-ohk

Thank you!

airocker2• 5 years ago

I wonder the Loki designers thought about the features that ES provides before rewriting the whole thing in a new way? Search and Analyze is pretty cool. Almost all of Loki can be an analzer plugin. Automatic schema detection is cool. ITs not that crazy to administer.

arminiusreturns• 5 years ago

I often find myself wondering if all this is really that much better than the old rsyslog/syslog-ng stuff we used to do. These days its all some timeseries on top of journald, has anyone tested the remote journald options? Also at what point do we break logs out from metrics? Trying to do everything in one tool is so un-*nix.

vimda3• 5 years ago

What we do internally is use syslog-ng (https://github.com/syslog-ng/syslog-ng) to read the journald socket and push to a remote and into Kafka. I think journald works well as a structured logging tool, but it's certainly deficient in other ways

reacharavindh3• 5 years ago

I have looked into ElasticSearch + Kibana as a solution to aggregate logs. There may be plenty of choice to replace ElasticSearch(ClickHouse, even Postgres, heck even journald), but a nice UI where you can simply search for that random piece of text you need to sift through the logs is the red herring.

Until now, I have not seen a web interface to log as powerful as Kibana that can work with anything other than ElasticSearch.

This is why I chose to stop my search and pay for Datadog to do this correctly, and simply allow me to search for that keyword on logs when I need it the most(and not worry about whether I indexed stuff correctly, or balanced some whatever in ElasticSearch, or remembered to setup something far too technical for a log system). Datadog allows you to keep a short periods worth of data in the index and "expire" old content into archives while retaining the ability to add them back to index if needed for any investigation.

chmike• 5 years ago

ClickHouse is written in C++ and is open source: https://github.com/ClickHouse/ClickHouse

dariusj18• 5 years ago

Where can I find resources on tools that integrate with Clickhouse, ex. are there any tools for gathering server metrics and sending them to Clickhouse?

rio5172• 5 years ago

Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.

vosper• 5 years ago

FWIW, the ES team implemented a change that I requested via GitHub, and I’m just some guy who’s not even a paying customer. The change involved the behavior of force merge, and probably didn’t affect too many people. It’ll allow my employer (and hopefully some other companies) to make more efficient use of our disk space.

I’m really appreciative of the way they considered and then implemented my request.

rio5172• 5 years ago

That's good to hear.

waste_monk• 5 years ago

I know someone who works at Elastic and it sounds like a really nice workplace (in terms of culture, average competency level across all employees, etc.), and that seems to be reflected in their products.

gertrunde• 5 years ago

I'm not sure I regard that particular example as actually bad.

The response of "use the client version that matches the server version" seems perfectly fair.

alkz4• 5 years ago

elastic 5.x is ANCIENT and no one in their right mind should still use that for any production workload

piaste4• 5 years ago

This is spot on based on our experience.

I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions.

We moved to NewRelic and while its dashboards are not _quite_ as fancy or powerful as Kibana, it's as close to zero-configuration as you can hope. It also has a bunch more features that show it's a monitoring system first and foremost.

Sure, it's a SAAS that we can't self-host, but diagnostic logs aren't business-critical so if we got locked out or priced out tomorrow, we would suffer no real disruption while looking for a replacement.

vimda3• 5 years ago

At an old job we used NewRelic right up until they tried to 3x our annual bill :/

devin• 5 years ago

You are not alone.

OldOneEye2• 5 years ago

At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used.

New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it would be enough. Not to ingest, that it did fine, but to query them under a reasonable time frame without a timeout, that's where it couldn't handle the high volume.

Also, their support service, while trying to win us over, that is, in their best moment, was nothing really stellar. Favouring sending us sales/presales people to solve technical problems.

It didn't leave us a good aftertaste.

pfarrell• 5 years ago

Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.

sateesh2• 5 years ago

I think for slightly lesser volumes Sumologic can be a choice. The kind of search queries, regex, and capture options give the feel of log parsing on a *nix box.

VectorLock3• 5 years ago

At high volumes, at my job, we have yet to find a good third-party log SaaS that performs

Not even just performs but the costs are always astronomically higher.

Sevii• 5 years ago

How are diagnostic logs not business critical? If you have an outage while your non-mission critical logs are offline what are you going to do?

piaste4• 5 years ago

If it's a live outage, we can still SSH into a machine and grep the console output / local rotating log files. (For that matter, I still prefer to do that when I'm just testing new stuff in dev/staging environments)

NewRelic "only" stores the logs for 30 days and displays them in a nice web UI with searching, alerting, sharing, and a bunch of other stuff. It's not like they cease to exist without it.

beardface4• 5 years ago

I believe the complaints here are a case of 'not using it correctly'.

The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place?

Full text search for logs is incredibly useful for log files when combined with alerting. If you get log entries indicating that a disk is full, a service has stopped, or a user account is blocked, Elasticsearch can (with the right license) send emails or post on Slack.

Static mappings can be a pain but if you're constantly increasing the maximum field count for an index, use different indices for different log sources. Come up with an index pattern or alias that allows querying all those indices at the same time.

The main task here is reconciling the different logs so the index mappings are easily searchable, effectively as a union. Elastic Common Schema helps a lot with this. Elasticsearch mappings are easier to build when you first consider the queries you're going to be running on the data. You can then design the mapping with the right structure, field types, and settings.

gpderetta• 5 years ago

I have little knowledge of the log aggregation domain, but generally indices are great for read mostly loads. It seems to me that for log aggregation writes are more frequent than searches; cheap writes and the occasional brute force search.

For alerting you might better off running each new line against a set of filters/watchers. It seems wasteful to run it after indexing.

Again, no experience or knowledge on the domain, so I might be completely off.

beardface4• 5 years ago

I have little knowledge of the log aggregation domain, but generally indices are great for read mostly loads.

Generally, you write and read to/from the same index in Elasticsearch. Where this falls apart is that you'll often want to change the configuration for an index based on whether it's write or read heavy. The main thing that changes in this scenario is the number of primary and replica shards (Lucene indices) for the Elasticsearch index.

Indices with a high write, low search workload will generally require more primary shards and less replicas. Low write, high search workloads require the opposite; lower primaries and more replicas.

The problem comes when you need high write and high search rates. Using a single cluster with lots of primaries and lots of replicas will overwhelm the hosts and you end up with terrible performance. The general pattern with Elasticsearch is to run two clusters. Index into one cluster, then use cross-cluster-replication (CCR) into a different cluster you run queries against.

There's an incredible amount of nuance to all of this. I've worked with many clusters and they all have different usage and configuration requirements. There's no magic formula for calculating configuration values; it all comes down to experience, monitoring, and experimentation.

cmarschner• 5 years ago

At the core of Lucene, as you index a document, it creates first an index containing a single document, and everything else is merge operations (operating in log N - merging larger and larger chunks). So the nice thing is that you can use the same query language, in fact the exact same implementation, to run a search query in alerting mode. You would create this single document index (which you'd do anyway to make it searchable) and run the query against it before adding it to the other documents.

vimda3• 5 years ago

My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case

nerdponx2• 5 years ago

This has been my experience. Obviously different people use logs for different things, but in my case I'm usually looking for information about something bad that already happened, within a very specific window of time, and within a specific section of the application. 99% of the time, that means I am filtering until there are only a handful of entries that match, at which point I don't need full text search at all.

willvarfar• 5 years ago

Completely agree.

My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields.

Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.

wardb2• 5 years ago

The not-indexing the log lines in Loki doesn't mean you can run complex queries on Loki. I've made a video to explain this concept: https://youtu.be/UiiZ463lcVA

jillesvangurp• 5 years ago

I can't say I agree with this article. Elasticsearch is a bit of a Swiss army knife and definitely needs some setup and configuring. But if you set it up properly, it does logs extremely well.

And it does actually come with useful defaults and well integrated tools. For example, if you use the various Beats agents to push data to Elasticsearch, they come with a default schema and out of the box dashboards that they can create for you. These are actually useful. I don't know that many tools in this space that have out of the box dashboards pre-configured, with matching schemas, etc.

Its also not that hard to create your own schemas and dashboards. Dynamic mapping is a rookie mistake that you might get away with if you don't have a lot of structured data. The classic mistake here would be mapping hundreds/thousands of fields. Not a great idea when you have billions of documents. The per document overhead adds up pretty quickly (disk, memory, indexing speed, querying overhead, etc). Much better is to only map those fields that matter to you and disable dynamic mapping for everything else.

And then there are features like rollups, data streams, index lifecycle management, etc. that make a lot of sense if you are storing logs at large scale. But you have to of course know to use them. As these were x-pack features, you won't be getting that with Opensearch.

Now there are of course many other solutions. But frankly it's a bit of a messy field of half integrated tools and a high DIY degree of how to set this up. Log analytics is only as good as the dashboards you get with them. Usually that means Grafana these days. How the data is stored is only part of the problem.

Grafana is basically an old fork of Kibana (v3) that has had lots of love and attention since then. It's a bit less feature rich perhaps but very capable. Incidentally, they went down the same re-licensing path as Elasticsearch: https://www.infoq.com/news/2021/04/grafana-licence-agpl/. So, you might want to think twice about putting that on a server unless you are completely fine having AGPLv3 licensed code close to where our own code is deployed (hint, if you are uncertain, you probably should not be doing that). LOKI is licensed similarly.

You can definitely gobble together some dashboards pretty easily using e.g. Grafana and any of its backends. But the out of the box experience for that is a lot of work figuring out what to use, how top operate it properly, etc. Elastic Cloud is expensive at scale but kind of plug and play on this front.

Probably the next best thing would be Opensearch. It's a fairly recent fork of Elasticsearch and Kibana. And unlike Elasticsearch/Kibana or Grafana/Loki, it remains under the Apache 2.0 license. Most of what you would need for a decent log analytics system is in there.

What you get by default from Google Cloud, AWS, etc. is pretty bare-bones in comparison in their cloud logging.

thinkharderdev• 5 years ago

My experience with ElasticSearch is that it is amazing for log aggregation until you reach a certain volume and then the operational cost (both $ and time/complexity) of running a large ES cluster starts to really bite hard.

To take one recent(ish) example from my last job. We built a bunch of streaming pipelines to enrich log data during ingest so we could use the ES ML jobs to do some proactive alerting on our application logs. It was all really nice but then we had to set up a regular patching cycle for our ES cluster. Generally everything is ES is stored in an index so doing a blue/green deployment works pretty well. Except that the ML jobs have some sort of in-memory state that can't be migrated, so the ML jobs would get migrated but in a weird, half-working state. So we ended up having to manually delete and recreate the "half-working" ML jobs whenever we rolled out the new patched cluster.

Having an "all in one" solution like ES is great until you hit a certain scale and the inherent complexity of such a system starts to really make itself apparent.

OldOneEye2• 5 years ago

Thanks for sharing your experience with ML.

We're experienced with managing a big scale elastic cluster, but never managed the ML jobs, so knowing about this limitation is definitely useful!

jcims• 5 years ago

Anyone that's used Splunk for any amount of time will feel like they are in a straightjacket with ES.

Anyone stuck paying for Splunk will think they probably should be.

dvdkon2• 5 years ago

AGPL is just GPL where usage over a network requires giving users the source code of the AGPL codebase. I'll take it over a custom licence where I have to worry if I need to release the source code of everything around it (SSPL) or if I'm providing it as a managed service (EL).

latch• 5 years ago

Kibana and Loki both load full messages in their list page. You end up loading megabytes (sometimes hundreds of megabytes) of data but it only displays a few kilobytes.

I don't know when we forgot the basic paradigm of list -> click -> details where loading the details is a separate http call. This is what datadog does, and the difference is staggering. Almost enough to make me abandon Kibana/Elastic and pay datadog.

I can't let a ELK discussion pass without mentioning vector.dev (https://vector.dev) which I'm not affiliated with aside from being a very happy user (for log ingestion).

est• 5 years ago

+1 for vector. Very stable, performing and feature-rich.

linsomniac• 5 years ago

Looks like Vector has a Loki outputter.

541• 5 years ago

Using ClickHouse for log storage and analysis is discussed here - https://news.ycombinator.com/item?id=26316401

jriddle567• 5 years ago

pure frustration and after a few hundred words it is about how cool the challenging software is. well it doesn't sove the problem, you just have some different problems on that scale. your problems require architecture

delaaxe• 5 years ago

As the name implies, it has been a great search system in my experience, although with a somewhat steep learning curve.

denysvitali3• 5 years ago

What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.

detaro• 5 years ago

(and I've seen this trend a lot)

any idea why? Seems like an odd approach to me.

denysvitali3• 5 years ago

Because people like to have everything (logs + metrics) in the same place. And probably Beats makes it easier to just ship everything in Elasticsearch (but of course this doesn't make it good)

atombender• 5 years ago

Because that's what Elastic is positioning Elasticsearch for. If you look at the features being added these last few years, so much is about time series stuff — aggregation, time-based partitioning, warm/cold storage, etc. Relatively little focus on structured search.

moduspol2• 5 years ago

I was interviewing earlier this year and saw at least three companies actively planning to migrate from their existing relational databases into Elasticsearch.

It seems to commonly be perceived as a panacea of databases.

closeparen• 5 years ago

When I see a nonzero value in a metric about “things that shouldn’t happen,” the very next thing I’m going to ask for is examples.

It’s generally considered too expensive to derive metrics from logs in this way, but it’d be a killer debugging experience.

beardface4• 5 years ago

Why do you consider this a bad thing?

101011• 5 years ago

I'm not convinced it is a bad thing. Good scale, relatively low latency, keeps your infrastructure costs down, and many of the other alerting options (looking at you Datadog and NewRelic) are crazy expensive.

Open to hearing other opinions though.

denysvitali3• 5 years ago

Because you're suddenly putting metrics (that should belong to a time-series database) into a full-text search database.

The result is a dashboard that takes ages to load just to show a trend in values.

I think it's using the wrong tool for the job, but maybe it's just me.

CSDude• 5 years ago

Elasticsearch, with a mapping that does not full text search indices disabled, and a relatively larger time, works very well for timeseries data. It does filtering & metric and bucket aggregations crazy fast. And it's very easy to add a new node. Just run on spot instances. We push event per request, which might have 100-500 unique keys, 100 millions of records per day, and no solution we've tried comes at the easiness of Elastic.

Try to replicate the freedom (no-schema, cardinality, adding a new node to cluster) of Elasticsearch with InfluxDB or others and you will hit cardinality problems real fast.

tunesmith• 5 years ago

I'm struggling with this currently. I want to record some metrics with our codebase, like counting each time a certain thing happens. Our code logs requests and response times to ElasticSearch, so there's the option of just "logging" these metrics so ElasticSearch has them too. It just seems a mismatch to me. Earlier I wanted to create a dashboard that graphed some derived stats off of that request data, like server utilization which depends on doing some math with average request count and average response time - it just didn't seem like elastic search dashboards easily supported that. I was able to do it on the AWS lb level with Cloudwatch. Not having had exhaustive training with ElasticSearch maybe I'm looking for a separate metrics system like statsd and Grafana.

stackedinserter• 5 years ago

It's fine for metrics, we've been using it for few years.

alkz4• 5 years ago

About the field limit, an application shouldn't have more than 1000 fields in their structured logging.

Usually when this happens is because the source uses what could be considered user input as keys, for example in access logs with stuff such as headers o query parameter, and you make a new field for each header or query parameter. In the latter case all you need to exhaust your available fields in the index is some nasty bot trying random query strings on your site. This can be easily solved in the ingest/logstash pipeline

It is true that you have to spend time to carefully map and process the documents you're ingesting in elasticsearch, but once you do I don't think there's any other solution with the same performance and features for logs, especially considering the licensed features (like alerting from anomaly detection machine learning jobs)

The elastic stack is complex and might be hard to grasp (their own training courses are a great help though), requires significant computing resources, and managing a big deployment can be a full time job, but once implemented correctly you can do awesome stuff with it.

Loki is a simpler and slower solution which does less things, so if you don't need what elastic provides, it could be a better fit as it's probably easier to manage

jurajmasar• 5 years ago

Agreed!

Ubers Clickhouse as a Log Storage thing

We built hosted ClickHouse-based logging as a service https://logtail.com, just launched with Show HN last week.

Disclaimer: I'm the founder, happy to answer any questions

exabrial• 5 years ago

It's not by itself. You need to put Graylog in front of it :)

OrvalWintermute• 5 years ago

I’ve really pushed ElasticSearch to its limits, with hundreds of terrabytes of data across dozens of machines and tens of thousands of shards and in all that time I’ve found that it really only works well for one of those situations.

This is a non sequitur

finnh• 5 years ago

Just use Scalyr.

chondl• 5 years ago

A million times this. Crazy fast flexible searching. I'm nervous they'll lose focus on the logging product after the SentinelOne acquisition but so much easier than anything else.

pachico2• 5 years ago

Elasticsearch is good because it just ingests whatever you sent to it, which allows you to deliver solutions rather quickly.

Having said this, I agree there are better solutions. (Also, Elasticsearch shines because of its full text search capability, which is not often exploited in case of logs.)

Loki is fine (or better said, it will be fine once they finally release a version without write-out-of-order constrain) but I find its lack of high-availability solution a bit frustrating.

ClickHouse, on the other side, is just magnificent. I use it in combination with Vector as message pipeline solution (it's an alternative to Fluentd, let's say).

So, yes, Elasticsearch is just not great and not only for logs, but for everything else that doesn't require full text search, in my opinion.

wardb2• 5 years ago

Out of Order support is available in Loki's main branch and included in the next release. It's already live in production on Grafana Cloud. https://grafana.com/blog/2021/09/16/avoid-dropped-logs-due-t...

High availability in Loki is supported in distributed mode. Helm chart here: https://github.com/grafana/helm-charts/tree/main/charts/loki...

pachico2• 5 years ago

Yes, I'll try the next release, that's why I said it wasn't released yet :)

Regarding HA, I meant something beyond a k8s deployment.

pytlicek• 5 years ago

Of course, it's not! It is a search engine and who knows its history will know it was not intended to be a logging system.

jmpman• 5 years ago

If I remember correctly, the core Splunk patents are from around 2006. Once the patents expire, could an open source project duplicate the Splunk indexing structures? As much as Elasticsearch has tried, Splunk is better for logs.

hintymad• 5 years ago

Grafana Labs' Loki is very exciting. Instead of storing a costly Inverted Index, Loki only indexes on fields (the equivalent of keyword fields in ElasticSearch)

One can configure Elasticsearch to index only on fields too, so I'm not sure "only indexes on fields" is a differentiating factor. The real advantage of Elasticsearch, or any search engine in general, is arbitrary boolean filter, as many log aggregation systems have started to use inverted index too. In addition, Elasticsearch has its own column-oriented data structure specifically for aggregation. Static sharding is a problem, but is not necessarily a big one, as many companies do not have enough scale to reach the problem yet.

BTW, we should really take a grain of salt on what Uber claims and what they do use. Case in point, they internally used Elasticsearch for years to aggregate all the logs in their marketplace for both real-time use cases and historical data that spanned months. Their Pinot-based solution and the promotion-oriented GPU-db didn't go anywhere.

annexrichmond• 5 years ago

I moved to a company that uses ElasticSearch as an observability stack. It has been such an awful experience. It’s just painful creating new dashboards and alerts, such that people avoid having to deal with it at all. One example is that alerts are completely independent from dashboards. It’s so difficult to get a full graph related to the alert condition to gather more context. The reason it’s like this, imo, is that they are trying to do too much with the same product and so each feature feels compromised.

At my last company I was very enthusiastic about monitoring where we were using Splunk and SignalFX. It was just fast, seamless, and reliable.

justinsaccount• 5 years ago

Uber has not open sourced this work so we are unable to benchmark it and see how it performs

I implemented their design here, specifically for importing zeek logs:

https://github.com/JustinAzoff/zeek-clickhouse

I don't have the elastic compatible query api though, or the smarts that auto materialize popular columns.

It works though, does a good job at soaking up any sort of log type and handling fields being added or removed.

siddharthgoel88• 5 years ago

Elasticsearch apart from being an overkill for logging, I feel has a high cost of maintenance as the system starts to scale. Somehow that cost is not very visible in the start. One of the Singapore based successful start-up (Grab) had blogged about some other pain-points a couple of years back ( https://engineering.grab.com/how-built-logging-stack ) as well.

chromatin• 5 years ago

The premise of the article is that it is unnecessary and inefficient to index the full text fields, as searching on fields is often more than enough.

Other solutions like Loki, which only index non-full-text fields, are suggested.

Why not RDBMS? I have over 10 years come full circle from SQL/RDBMS -> NoSQL of various types -> SQL/RDBMS.

edit: Is it really that other databases types are better suited to high ingest rate / write-heavy light-read workloads?

toddpersen2• 5 years ago

<shameless-plug>

Former co-founder and CTO of InfluxData here, currently building a new company in this space. My strongly-opinionated view on this is that Elasticsearch is not a time-series database and asking it to handle large volumes of logs (fundamentally a time-series use case) is always going to be painful and expensive.

We've built a product called EraSearch that mimics the Elasticsearch APIs for ease of integration but is built with a significantly more efficient (read time-series) architecture. We can handle ingest volumes with about 1/10th of the hardware required for Elasticsearch while still offering comparable (or faster) query performance. If you are generating large amounts of logs (~1TB per day or more), my guess is that this will resonate with you.

If any of this sounds interesting, drop me a note at todd@era.co - I'd love to hear more about your use case. Or even if you just want to talk about time-series data, I'm game. ;)

</shameless-plug>

rad_gruchalski• 5 years ago

I’d love to try your product out in combination with Jaeger. Is it possible?

toddpersen2• 5 years ago

We've actually been doing some internal work with OpenTelemetry and Jaeger - we would love some feedback on it. Drop me a note at todd@era.co and we can get you set up with a demo instance.

t-writescode• 5 years ago

About 5 or 6 years ago, my previous job introduced ElasticSearch to our infrastructure.

Our existing codebase had a pattern where all logs for a transaction were stored in a single, big log and then that log was uploaded to a server to be stored.

We moved this large log to ElasticSearch, formatted it differently, labeled a bunch of columns, used Logstash to standardize variable names, etc.

We did this for our main services and kept those services separated as different indexes.

Each log had what user did the web request, response codes, we had detailed logging and general logging stored in 2 separate places and threw out the detailed logs after a very short while. We had fields for all common detail work.

It. Was. Perfect. Everyone could use it. Our in-house customer support team used those logs to help diagnose customer issues, our tech team used it to track issues. Our NOC used it to investigate issues.

We had tens of dashboards that were on a rotating view. We had at-a-glance server health tracking through it, all from active traffic that was being formatted and used.

And then the next company I worked with that used ElasticSearch tried to use it in a world where each logger.info() was its own row in ElasticSearch. That just seemed like a horrible, horrible idea.

It really does seem to come down to how you use it.

lazyant• 5 years ago

ES and the ELK stack is the worst pile of shit I've had to deal with. Everybody I know who has worked with it has had problems. Obscure messages that nobody can figure out what they mean. Dropping logs silently, sometimes because it can't reconcile two index mappings. Constant baby-sitting and needing to add more hardware even if there are no indications that it needs it, super hard to tune or understand. A proper db doesn't need this much work and shipping logs (syslog) has been solved since the 80s. Move to Loki.

jakozaur• 5 years ago

(sameless plug of Sumo Logic employee)

The SaaS vendors are much better solution for logs, such as Sumo Logic. They have multiple tier of data (continuous, frequent, infrequent) which different performance and cost trade-offs. Most companies would be better, dividing logs into different category than running and tuning Elastic Search.

I don't think Elasticsearch is a good logging system

Related Stories