spimmy

ha! ok, i never thought of it quite like that.

just made this exact same point myself, lower in this thread. (author here)

yes! love this. and many great engineers will go back to being "good engineers" as they pick up new skills. rinse and repeat++

i think it has to start with having engineering managers and directors who are deeply technical themselves. the idea that you can split up this work into "managers do the people stuff" and "engineers do the technical stuff" is bananas. it's all sociotechnical work.

this is why i advocate the engineer/manager pendulum so strongly. we get better results when management has strong tech skills (and staff+ engineers have organizational skills as well).

well obviously there's a ton of variance here. but in the type of engineering i'm most familiar with, any sizable amount of production services or surface layer being owned by a single person is a bad thing.

individuals can get sick, go on vacation, etc. having it be owned by a team creates resiliency from a people perspective.

oh wow i missed that, thank you!

many fair points here <3

idk, avoiding landmines seems like pretty significant business impact

because there is, by definition, an extremely limited supply of the "best engineers in the world". if you can afford to pay top-10% salaries, and attract and retain top-10% engineers, more power to you.

maybe not literally "any asshole" can do it -- but it certainly asks more of your leaders to craft sociotechnical systems oriented towards learning and enablement. and i don't think that's a bad thing.

i'm struggling to see how what you are saying you value is any different from what i am saying i value (author here).

Word.

insightful; cosign. (i wrote the piece)

This is a good insight, but they already have a therapist. :) (I am the author)

Excited to see this!! Very much on board with this sort of approach to the future -- wide, structured logs, a balance between machines doing the heavy lifting + humans adding intent to the signal. Also VERY excited to see honeycomb customers already popping up in the comments .. i can't wait to see what these tools can do together.

jesus christ ok

it's weird that you think it's a sales pitch, when i ended it by pleading for other people to share writeups of their similar solutions. i know they exist, and i know people who are desperate to use them.

if it came across as a sales pitch, i def missed the target somehow, apologies.

HUGE +1 to mixing systems observability with product data. this is an oft-missed aspect of observability 2.0 that is increasingly critical. all of the interesting questions in software are some combination and conjunction of systems, app, and business data.

also big agree that most places are so, so, so messy and bad about doing logs. :( for years, i refused to even use the term "logs" because all the assumption i wanted people to make were the opposite of the assumptions people bring to logs: unstructured, messy, spray and pray, etc.

they've had nice things in BI land for YEARS. it's very cobbler's children have no shoes that we're still over here in software land doling out little drips of cardinality, guessing, eyeballing and jumping to conclusions. nice tools with nice data make alllll the difference.

software development should be a creative, curious, collaborative job... and it can be, with the right tools

i would love to hear how you are doing all the 2.0 stuff i described on datadog. you can't zoom in, zoom out, identify outliers and correlations.. the data doesn't exist! at best, you can predefine a few connective points between your logs and metrics and traces.

which is fine.. if your systems aren't that complicated and rarely fail in unpredictable ways. if that's the case, -- i'm glad you've found something that owrks for you.

my eyes popped at the "most expensive solution i've seen so far". compared to what?!? we don't like to promise "it's always cheaper", but .. it's always cheaper, lol.

with datadog, you have to arm wrestle them for every drop of cardinality, and on honeycomb, you can throw in as much as you want, any time you want. it smells to me like you aren't used to instrumenting your code with rich data?

i just think that metrics are the right tool for the job when the job is summarizing vast quantities of data.

not when the job is understanding complex systems. in order to do that, you need a ton of context and cardinality, etc. i know so many observability engineering teams that spend an outright majority of their time trying to skate the line between "enough cardinality to understand what's happening" but not so much that it bankrupts them. it's the wrong tool for the job. we need something much more like BI for technical data.

i'm not dismissing otel at all! under the hood, actually, everything is an event in otel ;)

hey, thanks! i would love to hear your feedback on how i could have made this simpler and easier to understand, if you have any. :)

What's wrong with banning politics at work?

news to me too, and it's my company. :)

i have no idea how these rumors get started.

^ CTO now, but thanks :D

Oh yeah, I wrote the blog post. And it had nothing to do with our launch. It was supposed to be funny, not a comprehensively detailed argument.

If you haven't been woken up by any or all of these things, I wouldn't expect you to find it as amusing as those of us who have.

It's modeled after the way Facebook does their own internal analytics, with Scuba.

Go ask them how they like it.

One company was paying $35k/ month in just ec2 hardware costs to run an ELK cluster, and we were able to slash that substantially, while still making money. Not to mention the savings in core engineering time.

Just goes to show you, people have no idea how muych their shit costs. And/or don't value their own time.

It's actually a lot cheaper. Honeycomb is a lot more efficient in the way it collects and stores data than any other option available. But yeah, side by side it looks worse. At this stage we are mostly talking to good engineers though, who understand how this works.