binarylogic

I agree with the framing. The goal isn't less data for its own sake. The goal is understanding your systems and being able to debug when things break.

But here's the thing: most teams aren't drowning in data because they're being thorough. They're drowning because no one knows what's valuable and what's not. Health checks firing every second aren't helping anyone debug anything. Debug logs left in production aren't insurance, they're noise.

The question isn't "can you do with less?" It's "do you even know what you have?" Most teams don't. They keep everything just in case, not because they made a deliberate choice, but because they can't answer the question.

Once you can answer it, you can make real tradeoffs. Keep the stuff that matters for debugging. Cut the stuff that doesn't.

Agree to an extent. There are absolutely unknown unknowns. But I think you'd be surprised how much data is obviously waste. Not the grey area, just pure garbage: health checks, debug logs left in production, redundant attributes.

That's why we break waste down into categories: https://docs.usetero.com/data-quality/categories/overview

But we don't stop there. You can go deeper with reasoning to root out the more nuanced waste. It's hard, but it's possible. That's where things get interesting.

Thank you! And you're right, it shouldn't cost that much. Financials are public for many of these vendors: 80%+ margins. The cost to value ratio has gotten way out of whack.

But even if storage were free, there's still a signal problem. Junk has a cost beyond the bill: infrastructure works harder, pipelines work harder, network egress adds up. And then there's noise. Engineers are inundated with it, which makes it harder to debug, understand their systems, and iterate on production. And if engineers struggle with noise and data quality, so does AI.

It's all related. Cheap storage is part of the solution, but understanding has to come first.

Thanks for the comment! Yes, I read that post. Great post. Feel free to reach out if you ever need help with Vector or have questions.

What you're describing is very real and it works to a degree. I've seen this same manual maintenance play out over and over for 10 years: cleaning dashboards, chasing engineers to align on schemas, running cost exercises. It never gets better, only worse.

It's nuts to me that after a decade of "innovation," observability still feels like a tax on engineers. Still a huge distraction. Still requires all this tedious maintenance. And I genuinely think it's rooted in vendor misalignment. The whole industry is incentivized to create more, not give you signal with less.

The post focuses on waste, but the other side of the coin is quality. Removing waste is part of that, but so is aligning on schemas, adhering to standards, catching mistakes before they ship. When data quality is high and stays high automatically, everything you're describing goes away.

That's the real goal.

You're right, it's not always binary. That's why we broke it down into categories:

https://docs.usetero.com/data-quality/logs/malformed-data

You'd be shocked how much obviously-safe waste (redundant attributes, health checks, debug logs left in production) accounts for before you even get to the nuanced stuff.

But think about this: if you had a service that was too expensive and you wanted to optimize the data, who would you ask? Probably the engineer who wrote the code, added the instrumentation, or whoever understands the service best. There's reasoning going on in their mind: failure scenarios, critical observability points, where the service sits in the dependency graph, what actually helps debug a 3am incident.

That reasoning can be captured. That's what I'm most excited about with Tero. Waste is just the most fundamental way to prove it. Each time someone tells us what's waste or not, the understanding gets stronger. Over time, Tero uses that same understanding to help engineers root cause, understand their systems, and more.

The question is answered in the post: ~40% on average, sometimes higher. That's a real number from real customer data.

But I'm an engineer at heart. I wanted this post to shed light on a real problem I've seen over a decade in this space that is causing a lot of pain; not write a product walkthrough. But the solution is very much real. There's deep, hard engineering going on: building semantic understanding of telemetry, classifying waste into verifiable categories, processing it at the edge. It's not simple, and I hope that comes through in the docs.

The docs get concrete if you want to peruse: https://docs.usetero.com/introduction/how-tero-works

My apologies, I fixed the link. So much for restructuring the docs the night before posting this.

You can read more here: https://docs.usetero.com/data-quality/overview

To loosely describe our approach: it's intentionally transparent. We start with obvious categories (health checks, debug logs, redundant attributes) that you can inspect and verify. No black box.

But underneath, Tero builds a semantic understanding of your data. Each category represents a progression in reasoning, from "this is obviously waste" to "this doesn't help anyone debug anything." You start simple, verify everything, and go deeper at your own pace.

Yeah, it's funny, I never went down the regex rabbit hole until this, but I was blown away by Hyperscan/Vectorscan. It truly changes the game. Traditional wisdom tells you regex is slow.

I'm surprised it's only 40%.

Oh, it's worse. I'm being conservative in the post. That number represents "pure" waste without sampling. You can see how we classify it: https://docs.usetero.com/data-quality/logs/malformed-data. If you get comfortable with sampling the right way (entire transactions, not individual logs), that number gets a lot bigger. The beauty of categories is you can incrementally root out waste in a way you're comfortable with.

compare logs from known good to known bad

I think you're describing anomaly detection. Diffing normal vs abnormal states to surface what's different. That's useful for incident investigation, but it's a different problem than waste identification. Waste isn't about good vs bad, it's about value: does this data help anyone debug anything, ever? A health check log isn't anomalous, it's just not worth keeping.

You're right that the dimensional analysis and pre-processing is where the real work is. That's exactly what Tero does. It compresses logs into semantic events, understands patterns, and maps meaning before any evaluation happens.

100% accurate. It is very much political. I'd also add that the problem is perpetuated by a disconnection between engineers who produce the data and those who are responsible for paying for it. This is somewhat intentional and exploited by vendors.

Tero doesn't just tell you how much is waste. It breaks down exactly what's wrong, attributes it to each service, and makes it possible for teams to finally own their data quality (and cost).

One thing I'm hoping catches on: now that we can put a number on waste, it can become an SLO, just like any other metric teams are responsible for. Data quality becomes something that heals itself.

Thank you for the nice comment. I'm glad you enjoy Vector. I poured myself into that software for many years. I'm a bit bummed with its current trajectory, though. We hope to bring the next evolution with Tero. There were many problems with Vector that I wished I could have fixed but was unable to. I hope to do those things with Tero (more to come!)

And yes, Tero is fundamentally a control plane that hooks into your data plane (whatever that is for you: OTel Collector, Datadog Agent, Vector, etc). It can run on-prem, use your own approved AI, completely within your network, and completely private.

Hey Peter, I absolutely remember you! Thanks for the nice comment.

And yes, data waste in this space is absurdly bad. I don't think people realize how bad it actually is. I estimate ~40% of the data (being conservative) is waste. But now we know - and knowing is half the battle :)

I spent a decade in observability. Built Vector, spent three years at Datadog. This is what I think is broken with observability and why.

Yeah, we're trying to solve for this over at https://vector.dev. We intend to decouple the pipeline from the method. As demonstrated by OT, this stuff changes, and you shouldn't have to rip out your plumbing every time it does. We're aiming to support all data (logs, metrics, and traces) as well as popular standards.

Hey Lemmox, good questions:

1) Not currently, but it's certainly possible. We don't foresee any major fundamental changes. The most likely change is around transforming. Specifically, how you shape both log and metrics data structures. Right now it's a little too piecemeal and we hope to provide ways to consolidate that more. For example, we have a transform for adding, removing, renaming, and coercing fields. We'll probably introduce a single transform to support that or think about ways to support schemas. 2) Yep! We're organizing our content and education strategy to start sharing this information. Expect to see high-quality guides, blog posts, and comparisons in the next 6-12 months.

Hope that helps!

I'm biased because my team and I created Vector [0], but I'd highly recommend investing in a vendor-agnostic data collector to start. You can use this to collect your data and send it wherever you please. This will afford you the flexibility to make changes as you learn more, which will be inevitable.

[0] https://github.com/timberio/vector

Ah, I'm amazed that I did not know this. I just updated everything to use MiB. Thanks for pointing that out.

They do, we actually put together a correctness test for this behavior:

https://github.com/timberio/vector-test-harness/tree/master/...

Logstash is not graceful. Our testing shows that they basically shut it down and start it again.

Absolutely, this is likely the next integration we'll be working on. There were a few features schema-wise that we needed support before we started, but we're _very_ close. We'd love beta testers to help us build it out. Feel free to email us if you're interested: vector@timber.io

Telegraf is nicely done. We spent a lot of time testing solutions in our test harness (https://github.com/timberio/vector-test-harness) and Telegraf was the most impressive of the tools we tested, so kudos to the Influx team on that.

But to answer your question, telegraf is very heavily metrics focused, and their logging support appears to be limited (reducing logs to metrics only). Vector is _currently_ focused on logging with an eye towards metrics, but still has work to do on the metrics front.

For example, we opened the door with the `log_to_metric` transform (https://docs.vector.dev/usage/configuration/transforms/log_t...) to ensure our data model supports metrics, but we still have a lot of work to do when it comes to metrics as a whole. Our end goal is to eventually replace telegraf and be a single, open, vendor neutral solution for both logs and metrics.

Happy to clarify further :)

It's very likely we're doing something wrong with this test, but after many hours of trying we couldn't get our simple test to pass for Logstash, even though it passed for others:

https://github.com/timberio/vector-test-harness/tree/master/...

Definitely open to feedback on what we're doing wrong.

This discussion is moot since Authlogic easily supports BCrypt:

http://www.binarylogic.com/2008/11/22/storing-nuclear-launch...

My comment about "anything not being used by the NSA" is not silly, it's true. Hell the NSA developed the Sha functions. I am not one to take security lightly, and salted Sha256 is not taking security lightly. I am not sure what kind of apps you write, but for 99% of the apps in the world, salted Sha256 is plenty secure. If it was as big of a problem as you said it would be highly discouraged, which it is not. The leading authentication solution in rails uses Sha1. Regardless, if you are extra paranoid, you can use any type of encryption you want with authgasm. So the argument is moot. Lastly, authgasm is not about promoting sha256, its about a style of authentication in rails. The method of encryption is just an option, which is what the crypto_provider option is all about. If you don't understand the subject which you are discussing, why are you discussing it?

I actually looked into bycrypt and it's great. But it would be silly of me to use a linux only encryption solution in a library available to the ruby public.

Also, thanks to github, if a plugin is popular enough it never really gets abandoned. Look at the engines plugin. The author of that plugin rarely does anything to it, but people fork it and keep it up-to-date.

Awesome, you shouldn't even consider using a plugin / gem unless it has a good name. The same applies with books. Unless you like title / cover you shoudn't read it.

I am the creator of this plugin.

Who cares about the name. I wanted something unique that would get people's attention and return unique content when searching. I also thought it was funny. People need to lighten up a little bit if the name bothers them. It's not like I put the name "authgasm" in helpers that you have to use all over your application. It sits in your plugin dir, that's all.

Anyways, regarding encryption. You can encrypt the password any way you want. Use bcrypt-ruby if you want. It would take no time at all to set up. Checkout the acts_as_authentic documentation. You can provide your own "crypto provider" and have it do whatever you want.

Lastly, I don't use Sha256 straight up. I add in salt. So reverse lookups are out of the question. Your last option is brute force. There is no encryption / hashing algorithm that is safe from brute force. Granted some might slow it down more than others, all that is doing is setting up more hurdles to jump, not 100% secure.

But I really like bcrypt, I'll look into it and maybe switch it to the default crypto provider. But, I really think for anything that is not being used in the NSA, Sha256 + salt would work just fine.