Get me out of data hell

https://ludic.mataroa.blog/blog/get-me-out-of-data-hell/

by pavel_lishin • 2 years ago

698 294 2 years ago

ludic.mataroa.blog

jaygreco• 2 years ago

I really like the author’s writing style here. The quips about the tea especially.

remram• 2 years ago

It reminds me of the classic series on /r/talesfromtechsupport by airz23. It's uncanny.

They would almost always feature a "I sip my coffee. It tastes like <feelings at the time>" Example: https://www.reddit.com/r/talesfromtechsupport/comments/203qx...

alex_suzuki• 2 years ago

Same. For me the association was the Bastard Operator From Hell, with a darker, more cynical view of life. Just very enjoyable reading.

ludicity21• 2 years ago

Very much appreciate the kind words! From you and the other commenters too.

ctippett• 2 years ago

of course, we're serverless, because how can you hurt yourself without a cutting-edge?

A beautiful epigram.

storafrid2• 2 years ago

Yes, but I'm surprised that they attribute "cutting-edge" to Lambda. It's about as old as Docker.

zdragnar3• 2 years ago

I was part of a company that went all in on using lambdas for the majority of their web facing APIs. That was 7 years ago.

The cutting edge bit is a nice quip, though I agree not exactly accurate anymore.

sph4• 2 years ago

It sounds like you like to deal with much sharper edges than I am comfortable with.

Or maybe I am too old with this shit. Still haven't found a use for "serverless" knives.

hinkley16• 2 years ago

For a bank, Lambda is brand new.

You haven’t worked in conservative industries I take it? Late adopters, every one.

OP is still trying to replace Cobol. I know an insurance company that started that process 15 years ago. Fifteen years.

Aeolun3• 2 years ago

It’s what everyone runs to when server based stuff would save them so much pain.

the_af5• 2 years ago

On one hand, yes. Beautiful.

But on the other, the same sentence could be written about software deployed to traditional servers. "Because of course, how can you hurt yourself without the joys of badly configured servers?".

skydhash• 2 years ago

You can hurt yourself with a badly held butter knife, and you can hurt yourself juggling katanas. Which situation would get people saying you're crazy?

hackable_sand3• 2 years ago

It's a pun.

layoric• 2 years ago

Came here to praise the same sentence, well done author, gave me a good laugh!

salt-thrower• 2 years ago

Beautifully written and fun to read. Blog posts like this give me a boost of mental strength to keep going during my worst episodes of burnout.

reverius42• 2 years ago

I've even degraded team morale because I've convinced some of the engineers that things should be better, but not management, so now some of the engineers are upset.

Oof, that hits a little close to home.

ludicity21• 2 years ago

The good news is that, since the others are also looking for work elsewhere, there will be more engineers out in gen pop that actually thinks tests are useful, hah.

hinkley16• 2 years ago

I have this illusion in my head that I stayed so long at my last company that almost all of my favorite people left, but one of my coworkers had my number.

After a person I liaised with on another team left, I asked his superior if there was someone else I should build bridges with. We started talking about one of the team members and he said, “I don’t want you to talk to him. We like him, and if you talk to him he’ll leave.”

This was on Slack so I don’t know if this was a jest or he was serious/mad. But it’s entirely true. I’ve convinced at least half a dozen people that we should expect better from a team environment and ourselves, and that this org (not the whole company, just this division) is a cult of stupidity.

I was trying to recruit collaborators to fix the bullshit but apparently they decided it would be much easier to just start over.

refulgentis3• 2 years ago

There's some finer points to parse in your comment that I'm not 100% on (ex. if "this org" is your division or the partner division), so I'm out on a ledge a little bit here, might not relate to what you meant.

I was lucky enough to get ~6 years running my own tech company after 6 years as a waiter. Then I sold it, yadda yadda, went to Google 6 months later, got ~7 years in there.

It really, really, really, disturbed me how approximately every situation, in every division, with any people, ended up being boiling down to "how do we muddle through one more day without challenging anyones preconceptions", 95% of the time it was tribal antisocial stuff, and no one would speak up about it.

Direct example, for posterity.

I don't wanna speak too directly to it, so lets imagine Google Division A (hereafter, dApps).

New division lead (ex-dApps) joins dBytes with apparent bias against partnering division (dConsumer). Despite the project being previously framed as top priority, new lead consistently undermines dConsumer in meetings and shows little interest in understanding their work. Team adopts leader's negative attitude, becoming obstructive and uncooperative. I ended up carrying a critical launch, virtually alone, for 6 months. At performance review time, my boss questions why I didn't get more team involvement - despite the hostile environment that prevented exactly that - and speaks glowingly about how we need to support peer going for promotion based on their excellent job on part Y...which they didn't do. They spent 2 days on it then said it was impossible. And they were definitively the most cooperative because at least they tried, and wouldn't actively be aggressive in meetings with the outgroup.

Everything, always, came down to: A) don't cause conflict at all, at home, or you will be buried B) we'll bend over backwards to accomodate conflict you invent, as long as we can clearly define them as an out-group with 0 ability to affect us day to day.

SynasterBeiter• 2 years ago

I hate whining like these. Just do your job and get over it. No need to be theatrical about it.

incognito124• 2 years ago

A creative outlet is a need, to some. You sound like OP's employees.

baq2• 2 years ago

Let the man live will you. Not everyone wants to be a drone for hire.

hobs2• 2 years ago

If this was a job with a non-abstract input and output process they would have OSHA saying it was genuinely unsafe (and definitely stupid.)

Many of us build systems to manage automatic actions to take care of this stuff, if I had an engineer I had to take 100+ steps to get something done I would definitely be considering 1) What the hell am I paying for and 2) Why the hell am I paying for it?

hprotagonist• 2 years ago

you know, he had a post about exactly this comment not too long ago!

edit: https://ludic.mataroa.blog/blog/the-violent-role-of-relentle...

tgv3• 2 years ago

That's pretty good. I'm somewhat curious now about his writing aspirations.

hipadev23• 2 years ago

Complacency is how this shit happens.

dangerwill• 2 years ago

Quick question: How many years of industry exp do you have? I thought the same way until this year when the burnout got to me too. I thought I was too much of a high achiever to get burnout and yet I'm in the same boat as the author.

Also, once the people who speak up about a problem leave, all you are left with are idiot yes-men in management, old timers doing their job as minimally as possible to not to be noticed by management, and fresh new engineering grads ready to be grist for the mill. When those sorts of people are writing all of the code around you, no matter how good you are, you will be driven insane.

brianhorakh• 2 years ago

Wonderful prose. I am in Melbourne also. I possibly used to work at the same place but I'm not sure.

I resigned due to the night terrors caused by the cyber security issues I saw everywhere. The more I explored and understood the more sleep I lost.

ludicity21• 2 years ago

Hah, yes, I'm not in cybersecurity but am very close to a few people that are. The incompetence is not evenly distributed and not as bad as it is in data, but some companies are in terrible states, and the stakes are much higher.

hinkley16• 2 years ago

I worked with an Aussie who was in the US on H1B because the aerospace industry was even worse than the status quo in Australia. Last I heard he went back. I sincerely hope he switched industry verticals.

zombiwoof• 2 years ago

Data “engineering” is where all the cool kids go with no clue and create insane architectures to justify their incompetence

hobs2• 2 years ago

As a data engineer I have seen absolutely bullshit pass for production, but it doesn't seem that different from all the other bullshit I have seen people deploy in my life.

It is one of the few types of jobs I have worked were someone credulously offering adding five more layers to fix an issue with latency is a normal operating procedure though.

parpfish• 2 years ago

data engineer gives data science a run for their money when it comes to ambiguous job expectations.

I’ve seen it mean anything from “distributed computing expert” to “knows SQL”

marcosdumay3• 2 years ago

to “knows SQL”

There are still ways to go. I've seen it means "spends the days filling excel spreadsheets".

sgarland• 2 years ago

knows SQL

If by that you mean, “knows the commands to create, fill, and select from a table,” then yes. If you mean, “knows how to create a performant schema and queries that will serve them well into the future,” then no, absolutely not.

OTOH, IME data folk are much more cheerful and willing to change things than devs when I point out the innumerable ways their DB choices are choking them. Devs more often fall on the side of “we don’t have time for that on our roadmap; can’t you just fix it?”

sevensor3• 2 years ago

“knows SQL”

Ah, clearly you’ve worked with a better class of data scientists than I.

WorldMaker4• 2 years ago

I've worked with people where "knows SQL" meant "knows the Access query builder UI, sort of, and demands the ability to query Production databases with the awful SQL auto-generated from the Access UI".

the_af5• 2 years ago

You can replace "data engineering" with "software engineering" in that sentence and it will still hold true...

fifilura• 2 years ago

Or chef or carpenter or automobile builder or priest or...

icedchai4• 2 years ago

I've seen some "data engineering" scripts that were complete messes and beyond crazy. Some examples: Massively over engineered "pipelines" that process a few hundred rows a day, but somehow manage to take forever to run. Developers that didn't know SQL beyond "select * from table", so they do all their summarization in Python. Or, worse, I've seen a Python script calling a shell script calling R calling something else, several more layers deep, when the same result could've all been done in SQL with a few temporary tables.

Oh, then I'm asked to "give this a code review before so-and-so does a deployment tomorrow." Uh, it's a little late to address any of the fundamentals, but there are hard coded paths everywhere...

bob10293• 2 years ago

I recently got a bit of a shocked reaction when I proposed to directly load daily files into temporary SQL tables and then use merge commands within the database to load the final tables. My use of code is essentially a shim between an SFTP client and SQL Server in this scenario. Maybe ~200 lines to connect, locate the files, run the bulk load operation, and then invoke the merge commands. Most of the fun bits are in the actual merge scripts.

Once your data is safely inside the database (temporary load tables or otherwise), there really isn't a good excuse for pulling it out and playing a bunch of circus tricks on it. Moving and transforming data within the RDBMS is infinitely more reliable than doing it with external tooling. Your ETL code should be entirely about getting the data safely into the RDBMS. It shouldn't even be responsible for testing new/deleted/modified records. You really want to use SQL for that.

You'll also be able to recruit more help if everything is neatly contained within the SQL tooling. In my scenario, business analysts can look at the merge commands and quickly iterate on the data pipeline if certain customers have weird quirks. They cannot do the same with some elaborate set of codebases, microservices, etc.

One specific thing that really sold me on this path was seeing how CTEs and views can make the T part of ETL 10000000x easier than even the fanciest code helpers like LINQ.

tgv3• 2 years ago

Some of my colleagues use Microsoft PowerBI, and indeed, they upload a few hundred rows of data (and a few hundred columns, which get unpivoted in powerbi to a say 40k rows). When they upload it, the powerbi instance overloads, and people get timeouts and such. That can last up to 20 minutes. I stay away from that as far as I can.

jitl2• 2 years ago

I wonder what company they’re describing here. It sounds like so many self inflicted problems that that you could undo or set right in a couple of weeks if you had the time and latitude to make changes across the system instead of being confined to a small area of team ownership.

bartread3• 2 years ago

I worked somewhere that had a lot of this sort of thing going on once. You cannot overestimate how hard it is to get anything done: politics and organisational dysfunction, not to mention that you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it, mean there are just huge scads of problems that, on the face of them, look relatively straightforward to solve but which, in practice, are organisationally impossible to solve.

whstl3• 2 years ago

Yep. I don't work with data engineering, but from hearing war stories from them, this could perfectly describe the last four companies I worked at. :/

you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it

I once sat down with a data engineer to try to fix a specific problem they had and that was 100% accurate. They were left to die by Ops and CISO.

Terr_• 2 years ago

A framing/question I like to ask use is: "Look for a root problem that ought to be fixed through a change in policy, politics, or incentives, and the wasteful use of time/money is how the company tries to avoid or defer facing it."

For example, Operations might demand that Engineering develops an increasingly-byzantine approvals process, to stop Sales from over-promising impossible or unprofitable projects.

deergomoo• 2 years ago

you probably don’t have access to half of what you need to in order to fix any given problem

I’ve found this to be one of my largest day-to-day problems even in a relatively functional organisation. Particularly when it involves something I can’t run on my own machine, like an AWS service.

In a previous role I often found myself constructing elaborate hypotheses about what was going on inside systems I couldn’t see into. I’d then need to try to verify it with someone on another team, in another timezone, who had the access but not necessarily the development background. Which usually meant getting on a screen share and asking them to click various things I wasn’t allowed to. If I was wrong, back to the drawing board and start again.

knallfrosch• 2 years ago

You're operating in the same company that let this happen in the first place.

And — not to take away from the piece – If noone noticed the logs have been garbage for two years, they can't be that important.

pavel_lishin• 2 years ago

The logs are unimportant until suddenly they're critical.

moregrist2• 2 years ago

It all seems easy to fix until you end up in a place where management process (and thus politics) has become more important than outcomes.

The people with experience and knowledge get pushed aside in favor of someone who can talk management-speak and (often) looks the role. Suddenly meetings where Things Get Decided only include managers who don’t know what they’re doing.

The best is when they make a mess and then get tremendous accolades for half-fixing it.

I’ve even seen it happen within relatively small startups. It’s a sign of rotting culture, but sometimes you have a mortgage to pay and have to get comfortable with the situation until you can find a better gig.

ludicity21• 2 years ago

I wrote a piece last year on how they were running a $500K Snowflake bill due to typing one number in wrong, and how I noticed within my first few weeks there by literally eyeballing something and going "queries take milliseconds so you really have to justify everywhere a minute appears in the configuration".

The logs are stupid (sorta) but imagine how many other issues can exist if we uncover something like this every time we open something.

taberiand• 2 years ago

Irreducible complexity and load-bearing bugs. Kludge built upon kludge built upon kludge. An engineer with the go ahead and all the support required would still struggle because the knock on effects of any one change cascade out in unpredictable ways. Not to mention working in an active environment where, although the other engineers support the goals in theory, they still need to deliver business requirements for a fickle management that doesn't truly understand what lurks beneath the facade - they don't have time to do it the right way, and if they try they'll just break both the old system and the new.

tiew9Vii2• 2 years ago

Sadly it sounds like most AU corporate companies I've worked at.

chubs3• 2 years ago

Why are we so crummy? Is it because our talent finds it generally easy to migrate to the states for better opportunities, because of no language barrier and the ‘coalition of the willing’ visa (iirc)?

lelandbatey2• 2 years ago

The typical problem is a culture where if you say

"I would like to make this systemically better by addressing our second order problems that are causing our very visible first order problems"

You will be told that we absolutely do not have time for that. The only actions you're allowed to take are fighting the fires closest to you, not turning off the pumps that spray the gasoline everywhere.

Typically this is only even possible because nothing that you're doing is actually used or scrutinized since if it was, someone would have immediately noticed that nothing works. Usually this is at places running on varying levels of investment dollar three-card-montey.

3eb7988a1663• 2 years ago

I inherited a pipeline like this. It is as if everything is a global variable. You cannot "just fix" one thing in isolation, because some spooky action at a distance of which you were unaware relies upon this insane behavior. Each and every hack is the expected input somewhere else in the chain. You have to carefully inspect everything downstream of any kind of minor adjustment because your cleanup is quite likely to break something else.

Immensely frustrating and draining where you can have accomplished ~nothing in a full day of work to fix what should have been a five minute change.

ludicity21• 2 years ago

I'm the author. You are exactly correct. Everything was so heavily interwoven that it was impossible to tell what would happen downstream without making an edit and then tracking the changes through dozens of steps through the architecture diagram.

zelphirkalt2• 2 years ago

Again and again it comes up in many scopes and contexts. Avoid global mutable state. But some people do not learn.

bartread3• 2 years ago

I’m going to read the rest of this. I’m enjoying it. But, simultaneously, part II has me so triggered - it bears striking resemblance to repeated situations I’ve encountered where the meaning and content of columns in a relational database were overloaded in varying degrees of heaviness (which is a practice I absolutely detest) - that I need to take a short break.

XorNot• 2 years ago

People do this with system hostnames a lot.

And it's almost impossible to get them to stop: the hostname should either be a random UUID or a random name from a pronounceable list depending on scale (or a syllabic UUID thing).

Because every other factor has one answer: you look up the other data you need in your CMDB. If that's too hard, you fix that so it's easy (DNS TXT records can be surprisingly useful here).

hermitcrab9• 2 years ago

I manage a database for a small local charity. I have set it up so that only I can add, delete or change the column structure. If someone wants a change, they have to email me and convince me (they are fine about this BTW). I'm sure the database would be an utter disaster zone by now if everyone was allowed to change it.

klysm2• 2 years ago

I think database schemas deserve to be protected with one’s life as the holy ground of the system. If the schema is fucked, everything else will be fucked too.

lifeisstillgood• 2 years ago

>> pretending that any of this is more important than hiring competent people and treating them well. I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets.

iamthepieman• 2 years ago

I do not use this term to refer to myself. I respect those who do and respect the meaning behind it but am just old enough that it feels alien to me 99% of the time.

But I am SO triggered by this piece. I had that intrusive feeling you sometimes get when driving where you think, "I could just close my eyes and see what happens", "Or that clif is so close and the guardrail doesn't really extend far enough"

Only for my career. Like I should just not show up on Monday. I should get in the car and drive far away and change my name and work at a nice retail joint in a mid-sized town.

I'm going to need to sit and stare into the distance for an hour and 3.

Electricniko• 2 years ago

There has been a point in my life where I ended every day in the dark, staring at a wall for an hour or two straight, trying to figure out why everything felt awful.

From his post about burnout and mental health. Also worth a read.

https://ludic.mataroa.blog/blog/on-burnout-mental-health-and...

SanjayMehta• 2 years ago

If on Monday morning you’re wishing it was Friday evening, it’s time to quit.

bryancoxwell• 2 years ago

I had that intrusive feeling you sometimes get when driving where you think, "I could just close my eyes and see what happens", "Or that clif is so close and the guardrail doesn't really extend far enough"

L’appel du vide

albert_e• 2 years ago

Does the mention of such concepts or acknowledging it is real ... put some lisetners (if they are work certain professions) under an obligation to refer the person to a mental health assessment?

Example: a blog post like this one, with the author's real name, that acknowleges it front and center: https://ebb-and-flow.blog/2023/07/23/another-scan-lappel-du-...

josephg3• 2 years ago

Seriously, quit then. It’s not worth it. You get one life. How many hours on this earth do you want to spend suicidally depressed? If you have a really high pain tolerance, maybe you can do that for years. How lucky would that be?

There’s a polish restaurant near where I live that makes amazing food. The owner is always out and about, chatting with customers and making sly jokes. Turns out he used to be an oracle sql consultant of some sort, and he turned it all in to run his restaurant. You can tell he’s thriving. I think he’s got the right idea.

andrepd2• 2 years ago

Survivorship bias. In an ideal world yes but in reality there's bills to be paid and tech (generally) pays really well.

Not saying you shouldn't quit, just that it's not so simple.

isoprophlex• 2 years ago

Change something for the better. You are the one who cares the most, you are the one best suited to take control over your life.

You deserve to feel good. Life is too short to be a cog in a broken machine.

strken3• 2 years ago

It's an almost exact copy of my last few months, right down to the 10am start.

Except that all our other senior engineers got laid off and there's nobody to pair with, I don't give two fucks about bullying because at this point the entire company knows I'll quit on the spot if they try, and our problems are mostly that the remaining team cannot understand the terrifying eldritch decision making process that led to fun little patterns like "wrap every API call in a try/catch and then ignore the errors".

I am seriously considering doing a TAFE course and becoming an electrician.

Buttons840• 2 years ago

They took inspiration from the error steamroller: https://github.com/ajalt/fuckitpy

GreenWatermelon• 2 years ago

wrap every API call in a try/catch and then ignore the errors

This describes the project I'm working on at $job with eerie accuracy :sob:

Dare I ask, was that project written in Java?

3923• 2 years ago

Will you feel better looking at a wiring job that seems guaranteed to burn down the building if you don't fix it? And will Management agree?

jauntywundrkind• 2 years ago

The observability world still regards itself as a system for monitoring, but reading (and sometimes seeing) how these systems just go so bad continues to drive a conviction that perhaps their strategies and tools should become bigger. That they should converge with business pipines.

We shouldn't just have wide events/big spans emitted... We should have those spans drive the pipeline. Rather than observability being a passive monitoring system, if we write code that reacts to events we are capturing, then we shuffle towards event sourcing.

Given how badly coupled together with shoestring glue & good wishes so many systems are, how opaque these pain zones are, it feels like the centralization upon existing industry standard protocols to capture events (which imo include traces) is a clear win.

(Obvious downside, these systems become mission critical, business process & monitoring both.)

halfcat2• 2 years ago

What’s this look like in practice? Is this something like business process modeling and workflow engines, or something else?

edejong• 2 years ago

Totally agree. Observability is just another dataset and should be modeled, managed and governed as other datasets. Data quality controls should be equal or of higher standard than regular data sets.

Monitoring, dashboarding and alerting should leverage other BI-class tooling.

halfcat2• 2 years ago

What’s the solution to wrangling these data projects?

The author’s experience is not far off from my own.

1. Any solution in place can only be understood by the person who created it

2. ”No, we can’t change that because then we’d have to validate everything from scratch again”

And therefore, as the author says:

”we'll continue with the work instead of fixing the critical production error”

I’m honestly not sure how to address it either. With traditional software dev we’d write tests, incorporate those into CI/CD, and start to course correct. We can use sample data to validate the code does what we think it does and that we didn’t break it.

But in these data projects, it’s not only the code that’s changing, but the data is also a moving target. You can write a test with sample data, but tomorrow your data might change because someone in sales added a custom field to the CRM, or IT upgraded the accounting software and all of the unique IDs changed, or someone upgraded their Excel version, or whatever.

And your code that works on the sample data needs to handle all of this, which obviously it can’t. You can try to validate the data somehow, check the schema, check if the number of rows hasn’t doubled or halved, and so forth, and then stop it from importing until you look into it, but also you can’t stop inbound data because an exec has a meeting in a few hours and expects their report to be updated.

I heard something about “data contracts” that’s supposed to address this, but it sounds like the next in a long line of buzz words intended to get management to buy another data product.

Has anyone worked in this kind of project that went well?

3np• 2 years ago

"We should add a single-source-of truth validation microservice that we put in front of all insertions, putting failing messages on a separate queue"

And on it goes.

Liquix• 2 years ago

https://xkcd.com/927/

ludicity21• 2 years ago

Author here, and also executive director at Hermit Tech now where we do things like this. Your approach has the core of how I'd go about it. The contract stuff is legit, though you don't need to buy a product for it.

The thing that is hard at big organizations isn't that executives need the data for meetings. The issue right now is that many organizations are already 3-4 years into building their analytics platforms, and Chief Data Officers worldwide are trying to prevent their role from disappearing. They're already very much "Mom, we have CTO at home" in many companies, as evidenced by the fact they're usually reporting to the CTO or CFO.

So at this stage, they've already told the business that the platform is "ready", and they are onboarding data sources. With no way to measure data quality, the only thing visible at the organization level is number of data sources onboarded. The fastest way to onboard data sources is to have good CI/CD and a solid developer environment, but this would probably result in slowdown for 1 - 2 months even if you had executive backing to bulldoze all objections from the IT department.

That's the sort of thing I can commit my team to as a business owner, but most executives don't have the nerve to slow delivery down and aren't losing money out of their pocket due to the inefficiency - I get to talk with a lot of them due to the blog's success these days, and many of them really are just employees with more status, with the same incentives. And to make it worse, the loss of nerve is actually understandable, because the type of team that would build something this bad will also waste those two months then still deliver slowly! But most people aren't thinking in terms this complex, and yes, I know it isn't that complex.

I'm expecting to pick up some work in this area at larger orgs in a few years when these leaders rotate out and new leaders rotate in and go "what the hell IS this?", but for now we're mostly aimed at helping smaller places do it right from day one.

ewuhic3• 2 years ago

Do you have a post describing the "right [way] from day one"? Spill your secret sauce.

pxc• 2 years ago

This blog post rescheduled all my appointments, tucked me in, sang me a lullaby, then woke me up with coffee and breakfast late the next morning. I am healed.

For real, a fun and refreshing read (if also a little haunting).

marcosdumay3• 2 years ago

The link about Scrum has a link to this:

https://agile2.net/

Can someone, please, tell me this is a joke. Because I can't be certain, but it doesn't look like one.

SSLy• 2 years ago

This is the real agile2 https://scaledagileframework.com/

marcosdumay3• 2 years ago

Oh, this is much worse satire. It didn't even put the effort to add details into The One Big Diagram.

(But yeah, this one I know is for real. The other one that feels like a joke, but looks real is harder to decide.)

ludicity21• 2 years ago

Author here. To the best of my knowledge, it is real. Sadly.

bobnamob• 2 years ago

Jeez, that’s almost as bad as the semantic versioning spec being semantically versioned

Muromec• 2 years ago

Great piece of writing from someone who truly cares about craft and suffers from the feeling that this craft is not what they are paid for.

Add: for people who sharer the feeling -- you can work in a place where velocity isn't all, managers are not assholes and you can dedicate yourself to craft.

baazaa2• 2 years ago

People always say this guy just has had bad luck with his employers but I live in Melbourne and work in data and reckon the whole industry is a scam.

Like why didn't anyone catch the issue with the logs? Because it doesn't matter, every data team is a cost-centre that unscrupulous managers use to launch their careers by saying they're big on AI. So nothing works, no-one cares it doesn't work, most the data engineers are incapable of coding fizzbuzz but it doesn't matter.

People always wonder why banks etc. use old mainframes. There's like a 0% success rate for new data projects. And that 0% includes projects which had launch parties etc. but no-one ever used the data or noticed how broken it was. I don't think a lot of orgs which use data as core-infra could modernize, the industry is just so broken at this point I don't think we can do what we did 30 years ago.

le-mark• 2 years ago

This jives with my experience at a financial services company. I once sat next to the “big data team” and the company 5 year plan was all about delivering analytics and ai to customers using their data the company housed.

The team consisted of one guy (who had a business degree) and a lot of empty cubes they were trying to fill. A year later the company had been acquired and the big data initiative had evaporated.

RangerScience2• 2 years ago

I have felt exactly this on regular full stack teams many, many times, so it’s also not just limited to data teams.

IMO a major factor here is that software engineering is both opaque and esoteric - at least with physical engineering, there’s something people can look at and think they understand.

baazaa2• 2 years ago

My theory is that data is worse again because at least if you're making a website you're expected to end up with a website. The process is opaque and esoteric, but the end-product is somewhat tangible.

A lot of data projects are moving and transforming data no-one cares about. They can fail completely silently, a manager can lie and say 'we've successfully built the data platform which is going to enable AI analytics' and it'll be like a misconfigured S3 or something. No-one's checking the end-product or even understands what it's meant to be.

tiew9Vii2• 2 years ago

Reluctantly worked on AU data projects for maybe the past decade. I don't classify myself as a data engineer, in fact I hate data engineering or data related work which is glorified ETL and SQL most of the time. They are the worst, broken projects I've done, not software engineering in the software engineering sense. I don't think I've worked on a good one yet despite the potential to be really interesting projects. I prefer general software projects doing a bit of everything as a generalist, data pays the bills though in AU.

Not seen/heard of this person before but reading this specific blog post it all sounds very familiar, it's depressing.

The "CTO" getting on stage taking a bunch of credit and everything being a mess or incomplete or lies is very familiar. Maybe not CTO but higher management. It's all smoke, mirrors, optics, self promoting, it works as these people end up making their way up the ladder when the lonely dev trying to do better work is just a dispensable cog in the wheel.

6510• 2 years ago

Not seen/heard of this person before but reading this specific blog post it all sounds very familiar, it's depressing.

Someone once told me he, as a form of therapy, rewrote the company he worked at in a few weekends. He never mentioned it to his coworkers, it was strictly a therapeutic effort. They apparently spend years "fixing" things without making any progress.

lelandbatey2• 2 years ago

I have seen data work well, but it only worked well in a situation where we had management focusing on two very tangible things that even the CEO could verify (since the CEO did know the product). Those tangible things were:

1. Accurate, auditable billing down to per-chargable entity/event

2. Dashboards for each customer that reflect THE SAME numbers as we generate in #1, so customers could see relevant info quicker than just waiting for the bill from #1

The only reason those things were valued and made a focus though was because a HUGE customer threatened to completely drop our company because that customer did an audit and noticed that we had overcharged them 3% because we were actually billing them on estimated numbers. That led to our CEO being personally yelled at by a much larger CEO, and our CEO (to his credit) didn't blame us (we'd raised the alarms that the bills were estimates and not auditable) but did say "this can never happen again, I trust you, do whatever needs to happen to make sure this never happens again."

And once we had #1 solid and tight, we were able to leverage that solid auditable data to generate solid dashboard numbers that always squared with what showed on the bill.

RHSman23• 2 years ago

A CEO worth working for

denimnerd422• 2 years ago

I lead a team on a large data project at an enormous bank, hundreds of devs on the project across 3 continents. My team took care of the integration and automation of the sdlc process. We moved from several generations of ETL applications (9 applications) netezza/teradata/mainframes/hive map reduce all to spark. The project was a huge cost savings and great success. Massive risk reduction by getting these systems all under 1 roof. We found a lot of issues with the original data. We automated the lineage generation, data quality, data integrity, etc. We developed a frame work that made everything batteries included. Transformations were done as linear set of SQL steps or a DAG of sql steps if required. You could do more complicated things in reusable plugins if needed. We had a rock solid old school scheduler application also. We had thousands of these jobs. We had an automated data comparison tool that cataloged old data and ran the original code vs the new code on the same set of data. I don't think it's impossible to pull off but it was a hard project for sure. Grew my career a ton.

RHSman23• 2 years ago

What was the main reason for your success?

d0gsg0w00f• 2 years ago

I think the difference is that technical and business leadership at a bank understand that data is lifeblood. Bad data will get you on the front page of WSJ and a phone call from a regulator in Luxembourg.

For a lot of smaller Internet companies, data is just a fluffer. The real business is in image and which VC bbq you get invited to.

neeleshs• 2 years ago

This story is more an exception than norm.

I know startups that hired data engineers, deployed warehouses,DBT, a BI tool and churned hundreds of reports, and in one case their DBT project has hundreds of files. No one in that company knew why any of it was used.

All said and done the business users wanted three reports.

More often than not data teams are self-serving than anything else.

bob10293• 2 years ago

I don't think a lot of orgs which use data as core-infra could modernize

I argue this is a happy conclusion, not a problem to be solved.

What would "modern" bring to a bank except even more pain & suffering? Database technology invented in the 80-90s is more than sufficient for tracking information at the scale that 99% of financial institutions operate at today.

Virtually every core conversion project I've ever heard of has been a failure or is currently a burning wreck on its way to the bottom.

The only new bank projects that touch data and seem to succeed are LOB apps with highly curated experiences that are tightly integrated with the actual front/back office business. Having buy-in from staff regarding your UX is way more important than spinning out a 20 page AWS architecture diagram. The CTO can only take you so far through the vendor approval process at a bank. Retail operations (i.e. the people who are responsible for the brick & mortar branches) typically have substantially more pull in these organizations.

makeitdouble3• 2 years ago

What would "modern" bring to a bank except even more pain & suffering?

In the most simple term, a future.

Except if your bank is literally too big to fail, at some point you have to either move on from 80s technology or at least bring in an adaptation layer, because your profit center have also moved on or you're facing harder competition.

A typical example is banks getting merged: there will be a fight to see which system stays and which one disappears. If you froze your technology 4 decades ago it won't be your stack winning. [0]

Another is the evolution legal frameworks: EU countries passed laws requiring interoperable APIs to perform standard banking operations. Being a customer of a decent bank or a fossilized one made a huge difference and the market grew a lot more competitive. People would start hedging their bets when legacy banks looked too far behind.

[0] The most interesting and recent example of this is Mizuho bank just miserably failing at that task to the point the gov. intervened and anyone not married to them probably moved out.

https://www.mizuhogroup.com/news/2021/06/20210615_2release_e...

8n4vidtmkvmk• 2 years ago

Yes, please, fix the UX. That was my biggest gripe, working as an FSR at the bank.

One particular thing was we had to convert transit #s into branch numbers regularly. We did this by looking at a sheet of paper of course. Eventually I got fed up and wrote a web app so you could just punch in the numbers and have it instantly convert. I checked and people are still using it 10 years after I quit, which means nothing has changed and they're still using the same god awful software.

They did move some data at some point. I know this because they screwed that up too and partially merged my mom's and my bank accounts, which is a pretty bad error. Would be worse if it was some rando. Speaking of... That's exactly what AT&T did.

chaxor• 2 years ago

What would "modern" bring to a bank except even more pain & suffering?

It probably depends on what "modern" means here. If updating from tons of COBOL to {Julia, Python, Rust, or some other well known language} with an update to an SQLite backend (or perhaps postgres is acceptable for very specific scenarios), that is likely a good choice due to being able to fix old cruft and add maintainability for the future. If it's a switch to some nosql database backend with everything switched to some cypher-based lang or anything that touches javascript in any way, it's probably a mistake.

__turbobrew__• 2 years ago

I would like a bank which supports U2F.

atoav• 2 years ago

The problem is that most orgs seem to do the wrong thing, because the incentives of the higher ups don't align with ehat is good for the org.

E.g. if you are a bank ideally you'd like all your processes automated and streamlined with extremly transparent data flows etc — and you want as many of the banks employees to be proficient in these systems and constantly work on improving the systems within an controlled environment.

In practise this is not the kind of thing that allows single managers to come across like heroes — so it doesn't happen that way and you get island solutions with duct-taped connections between.

ludicity21• 2 years ago

I also get to spend a lot of time with executives thanks to the blog's success, and part of it isn't just incentives, it's pure confusion. People have no idea what they're buying.

I also get invitations to "sponsor events" now, since people see "director" on LinkedIn and think I have way more money than I do. Their business model seems to be flattering executives by inviting them to events where they can network with other rich people, then ask me for "sponsorship" money so that I can go into the room and brainwash them with my marketing material. I might even try it at some point to see if that's an accurate read.

ludicity21• 2 years ago

Author here. I now know some places in Melbourne that have a good success rate on projects. Some of them are so small as to be invisible and rarely hire. One uses two specific independent recruiters or internal referrals. As far as I know, they are extremely profitable because the competition is a joke.

For many organizations, the success rate is indeed 0%. A Group of Eight university (our top 8 universities nationwide), for example, sent me a job description a few months ago where they misspelled the word engineer, and left change tracking on in the Word document. This allowed me to walk through the profiles of the people running their data projects, and it was super obvious that many of the people involved aren't going to do a good job. They could have saved millions by having a random HN person eyeball the CVs of their chosen leadership team.

svilen_dobrev• 2 years ago

hey. Congratulations on your decision.

i think it all goes deeper in overall culture/attitude there.

i was in Melbourne in 2012.. with idea to relocate wholesale, 2nd time. Worked 2 months at some "startup", that fired me when i finished the task given.. Seems it was cheaper to hire "permanent" then fire, rather than take someone on 2 months contract. So that's one red light on the dashboard.. There were other redlights from overall "society", feeling something-is-wrong but i did ignore them for quite a while - people have become evil, etc..

Then i started going around places and mailing my cv here or there (with 22y of experience making software, by that time),.. ibm, ernst&young, you-name-it.. to no avail, and more red-lights flashed on me.. And one day visited some kind of meetup, organised/held in some wellknown company. Seemingly it was kind-of "hiring" event or so, we grouped in teams of 3-4-5 people, with half from company, and other half outsiders.. and went solving some problems of theirs. Or that was the "label". Any solution that any of outsiders suggested, was shot down, with somewhat vague reasons, that at the end started to sound like "if we solve this there'll be no job tomorrow". And Smile :) Lots of smiles. Empty ones.

That was one of the Last red lights on my dashboard. Whether it was a financial balloon pressing everyone so they only smiled and did nothing, in order to pay the mortgage, or something else, i don't know. Next day i watched Sacrifice/1986/Offret by Tarkovsky, and.. bought a ticket out. Discontinued my oz dream. For good.

quoting meself, from 2007-8: "with time, places change people. Other way happens noticeably only while coming in - or switching on."

have fun

nyarlathotep_2• 2 years ago

> I live in Melbourne and work in data and reckon the whole industry is a scam.

You needn't live in Australia to reach that conclusion.

chubs3• 2 years ago

Do you think perhaps the problem is rooted in people being dishonest, and honest people are driven mad by it all and self-select out? The dead-sea effect?

ludicity21• 2 years ago

It's so many things. Dishonesty, lack of technical competence, political pressure, hype, organization structure, and incentives.

If I had to summarize though, it's that the median performance in any field will be at much lower levels than outsiders expect, and some fields with hazier results have this level set very, very, very low, especially when they're hyped up. But also that the market is actually at least a little bit efficient, but over long time scales. I think there's a 50%+ chance that the role of Chief Data Officer begins to die off, but also that it'll be replaced by something silly.

DeathArrow• 2 years ago

Because it doesn't matter, every data team is a cost-centre that unscrupulous managers use to launch their careers by saying they're big on AI.

If you have lots of data flowing and have full teams "working" 24/7 on it, does it really matter if that data is junk and that is not processed in a meaningful way? You can still ask AI to generate some nice looking charts with big numbers to show to investors. Investors like nice charts and big numbers. Or so, some businesses people think.

But in all reality the investors will ask questions like: how will this solve problems for customers, how do you intend to sell this to customers, how much does it cost, how will this generate me money. Unless those investors plan an early exit by finding other, more gullible investors than them, kind of like knowingly investing early in a Ponzi scheme.

CalRobert• 2 years ago

Ten years ago data engineering was another discipline in software engineering, like backend or frontend. Somewhere along the line the term was co-opted by “I can maybe barely string together some untested airflow pipelines” and it means something much different now.

WorldMaker4• 2 years ago

It is said that every major, still living COBOL program contains a bespoke, (sometimes poorly optimized) database engine with no standard query language, the only query tool was more program code. Perhaps the longevity of Mainframes points to there was some wizardry/safety lost in standardizing databases, giving people the impression that data itself was standard and too many tools to footgun data into foot pain, that we lost when databases were defined entirely as COBOL internals?

(Not that we haven't gained a lot from modern database tools, just something to think about that maybe the data siloes were good sometimes, too.)

akdor11542• 2 years ago

Agree.. I can tell you at least one Melbourne-based Flybuys retailer calculates your points with an unholy daily-scheduled stored procedure in Snowflake SQL, because.. big business dysfunction reasons lead to the data team being assigned to do it, and the data team didn't actually have any software engineer roles in it.

At least it has tests.

jpmoral4• 2 years ago

Do you mean the points earned in-store that are then sent to Flybuys to add to your total? Or, god forbid, do they do their own total?

photonthug• 2 years ago

it doesn't matter, every data team is a cost-centre that unscrupulous managers use to launch their careers by saying they're big on AI. So nothing works, no-one cares it doesn't work

Yes. Lots of times the most important asset for these companies is actually contractual obligations in terms of exclusive access to data or customers. It doesn’t matter if the product works, you’ll have to buy the company to build a different one that does. But the (broken or nonsensical) product pushes up the value of mergers and acquisitions. If leadership completely makes shit up then they might go to jail, so, they burn X million on “work” and cloud spend as part of an elaborate argument that it should sell for 10X.

the industry is just so broken at this point I don't think we can do what we did 30 years ago.

Well no, it’s never been easier to do high quality engineering, but mbas are in charge. They don’t think like philosophers or scientists and don’t traffic in common sense.

For anyone questioning their life / career choices because of this, it’s not about you. An individual working in an environment like this can still be a craftsman of integrity if they focus on small problems and solve them well, but you need to be able to get satisfaction from that, not from some overall mission (which again, is probably fake). If you’re most motivated to work directly on architecture, unification, etc, and want to change lots of things then you will probably be miserable.

But if you’re feeling shitty about the whole thing, it might help to realize that the actual nuts and bolts of adtech/martech data pipelines are much the same as the ones for cancer research or particle physics or climate science, so one can at least try and get transferable skills if circumstances are currently holding you hostage. Data isn’t a bullshit job. Leadership and management that just want to play games is the problem.

neumann• 2 years ago

100% agree with most of this. Most large organisations in Australia are just clueless about their needs and ride hype cycles with execs acting on FOMO. And they bought the lie that you need all these separate distinct engineers who are professionals in a niche. They think they are building a high end kitchen by having specialists and think they are hiring sous-chefs etc, but most of the time they are hiring line cooks, and in fact most of the time they should just be hiring short order cooks.

Most teams and businesses I know who are doing great things with data are mostly smaller companies and tech built up start-ups and hire one of two types of people. Generalists who can fill the gaps and are interested in everything, and staff software engineers who are amazing developers and asked to put together data pipelines with consideration of the whole infrastructure.

When I was looking for work, all the good companies were not hiring data engineers or machine learning engineers. They were hiring Senior Software Engineers with a remit to build data infrastructure, or build machine learning models. And they immediately removes 90% of the noise applicants.

fphhotchips• 2 years ago

Melbourne is easily the worst city in the country for this. Most of the tech sector is in the very large enterprise space lead by the banks, and as a result it's who you know and whether you went to Melbourne Grammar or Geelong Grammar that will determine which company you work for once you reach a certain level. Sydney is better just because there's more smaller stuff going on, and because CBA is better than NAB and ANZ combined on tech. (I hate Sydney otherwise and am based out of Melbourne)

Some places in Melbourne get real work done, even in the data sector. They're hard to find, but they exist.

djoldman• 2 years ago

This kind of thing is pretty typical.

The older the company is, the more likely one finds this morass.

It won't change absent powerful technical leadership.

holden_nelson• 2 years ago

I went down the rabbit hole of this blog after reading this post. This person's blog is amazing. I particularly appreciated this piece: https://ludic.mataroa.blog/blog/quitting-my-job-for-the-way-...

sph4• 2 years ago

"and still you will lay upon your death bed thinking that you may fend off the Reaper if you could but Estimate how long dying will take!"

What would I give to be able to compose sentences like this one.

sevensor3• 2 years ago

My man has a podcast as well, which is worth a listen.

ludicity21• 2 years ago

xoxo thanks! Although editing audio is one of my daily humility reminders. It's touuuugh.

snidane2• 2 years ago

Looks like the classic mistake of every data team. Every single office person works with data in one way or another. Having a team called 'data' just opens a blanch check for anyone in the organization to dump every issue and every piece of garbage to this team as long as they can identify it as data.

That's why you build data platforms and name your team accordingly. This is much easier position to defend, where you and your team have a mandate to build tools for other to be efficient with data.

If upstream provides funky logs or jsons where you expect strings, that's for your downstream to worry about. They need the data and they need to chase down the right people in the org to resolve that. Your responsibility should be only to provide a unified access to that external data and ideally some governance around the access like logging and lineage.

Tldr; Open your 'data' mandate too wide and vague and you won't survive as a team. Build data platforms instead.

_jonas• 2 years ago

I'm excited for LLM applications that can setup, monitor/validate, and optimize data pipelines at scale. Seems possible soon given that SQL and most data records aren't intended to be human-friendly

FridgeSeal• 2 years ago

Once again. Because clearly this needs to be said loudly and repeatedly.

That is a _technical solution_ to a fundamentally _social_ problem.

sph4• 2 years ago

Ah, the hubris of youth.

spencerchubb2• 2 years ago

When LLMs can do the following, they might be able to fix data hell:

- Negotiate with different teams to figure out what a field means

- Be told that a field should be converted from one format to another, but oh wait it's causing errors somewhere downstream because it was told the wrong instructions

- People come to you with some issue about the code you maintain, and you dig enough to realize the root cause is another team's code

hinkley16• 2 years ago

- explain how they came up with “12” as the answer.

hermitcrab9• 2 years ago

And now we have two problems...

hinkley16• 2 years ago

Ten problems. Regex is for wusses who fear danger. LLMs are where real men turn.

sourcepluck4• 2 years ago

Either this is a joke comment, or you haven't seen https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...

codethief• 2 years ago

Largely unrelated, but "data hell" reminds me of this classic from Silicon Valley: https://m.youtube.com/watch?v=YPgkSH2050k

hinkley16• 2 years ago

“I don’t want to live in a world where other people make the world a better place better than we do.”

Mike Judge did such a great job of making that character into Dollar Store Steve Jobs.

lucidguppy2• 2 years ago

because we don't have any tests,...

Right there ^^^

lucidguppy2• 2 years ago

I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets. It would take me a month tops.

^^^ Then do it... and then strangler fig the original.

ahoka• 2 years ago

You can’t as it will immediately draw out the leeches who will hurl “best practices”, strategic partnerships with scam software vendors and compliance check ticking bullshit at you until you back off.

ludicity21• 2 years ago

Author here. This is essentially the case for many clients, especially governemnt. But we're bootstrapped, so we can afford to consult and use what we've learned to build tooling that only targets a small handful of sane clients.

I've sought advice from various people on this, some who are famous-ish or quiet sales powerhouses in the US. My question was "What are executives buying when they hire consultants?", and the answer is consistently "Comfort". No one is actually comforted by Deloitte, KPMG, whoever.

The moment I had confidence in an ethical consulting practice is the moment someone said "I don't even know where I would hire good consultants". This was someone with 30 years or something absurd of industry experience, including mentoring people that went on to become staff engineers. After processing that, I realized I don't know where to hire a consultancy that isn't going to bait-and-switch me with mediocre talent. They obviously exist, but they probably can only support something like 1 to 10 clients each.

Even Thoughtworks, a place that I used to hear mentioned positively, was flagged by the CTO of a >$1B company over lunch last week as "shifting to bait-and-switch" tactics.

tl;dr Pretty sure you can absolutely do better and make a living off that, but you have to think very carefully, do research, read a bunch of sales/marketing books, have great communication skills and at least adequate engineering skills. I still don't know if I have some of those, but if I fail it'll be a skill issue, not because the problem is not tractable.

erulabs• 2 years ago

As a regular old “platform engineer” I fight to ignore “data platform” tasks. There’s no target to hit, it’s just moving sand around a sand box.

If you want an answer to a specific question, we can spin up a read replica and a Metabase and write a query in an afternoon, cool. I’ll get you a chart, we’ll move on. If you want “a data analytics platform to enable blah blah blah” I’m out, I can’t do it. My eyes won’t focus, my hands stop moving.

Developers sometimes tell me stuff like “Kubernetes is too complex”, “jeez React is a pain”. I send those quotes to my friends stuck writing 195 step DAGs to transform log files from s3 into s3 so they can eventually land in s3 - ah yes but they’re parquet somewhere in between, and that matters for some reason. We laugh together, but I can see it hurts them more than I intended.

Life is too short to faff about doing nothing. Go join a company with less than 100 engineers and learn to be happy again. Let the enterprises burn, we’ll all be better for it.

Anyways this was a fantastic piece, I hope this person writes their book after all.

spencerchubb2• 2 years ago

my friends stuck writing 195 step DAGs to transform log files from s3 into s3 so they can eventually land in s3

sounds similar to what I do! on my team, it does actually serve a purpose, it's just hard to see the forest from the trees sometimes

hermitcrab9• 2 years ago

Go join a company with less than 100 engineers and learn to be happy again. Let the enterprises burn, we’ll all be better for it.

But but then I won't be able to lease a new panzerwankmobile every 2 year to impress people I don't like!

hinkley16• 2 years ago

In police dramas there’s always the judge who refuses to sign a warrant and accuses the cops of going on a “fishing expedition”.

That’s half of Big Data to me. Maybe more. No idea what we’re looking for but we will tell you after the checks clear.

LAC-Tech• 2 years ago

This has such strong Australian vibes.

AU dev scene is not great. Really heavy with POs, and PMs and CTO's without the background.

0xbadcafebee2• 2 years ago

I've worked for like a dozen companies full-time. Most people don't know what they're doing. I always thought 'impostor syndrome' was a projection of general insecurity. But I've started to think it's actually the subconscious saying "I'm not sure what's right or wrong, please consult an expert."

I have a fantasy of quitting my job to write books on the [modern] theory and practice of information systems engineering. Not 'how to write software', that's been done; I mean all the forms of engineering around software/information systems. In my dream, I write the books, everyone reads them, and starts doing their jobs right.

But then I remember, I, a person arrogant enough to believe he knows how to do things right, still can't get shit done right. Maybe if I were a one-man company, I could 'do everything right', and feel good about the result. But I depend on an entire company of people to do the right thing, in the right way, at the right time. That's hard even with the best people. No company is made up of the best people. It's always a mix of the best, worst, and in-between.

Strangely, a company can put out a decent product, despite the company being a tire fire. This is some comfort when you get older. You realize that everything being shit is okay, as long as the bills are paid. I have PTSD from when the thing that paid the bills was on fire, every week, for years. Lately at every job I have, I internally panic and scream at how horrible everything is. Because I'm haunted by what might happen. But it's not happening yet. So I muffle the screams, smile and nod along with the stand-up-meeting-cum-status-update.

The sad thing is, I forget that it's okay that the stand-up is shit. I forget that I'm still getting a fat paycheck just to sit in meetings that could have been an e-mail. I forget that, despite the company bleeding cloud costs [no savings plans, RIs, serverless, right-sizing, etc], we seem to be making a profit. Despite the terrible designs, bad process, ineffective leadership, absentee management, lack of security, and all the rest, the bottom line is fine. The shit is fine. Currently, and probably for the unforeseeable future.

I get craftsmanship. I'm a crappy woodworker. I enjoy making things well, and getting better at it. But our jobs are not fine woodworking. Our jobs are construction. We are banging rusted nails into shitty, twisted, racked, cupped, knotty-ass studs. If we're lucky. Yeah, this building is going to be shit. But somebody's still going to pay for it. And there'll be another job after. If we really wanted fine woodworking, we never would have taken this job, and we know it. We'd be struggling to sell a cabinet that took us two 80+ hour weeks, too tired to appreciate its beauty, too defeated by flaws only we notice.

So let's stop beating ourselves up. Let's stop beating each other up. We don't, can't, won't, find meaning in this monument to mediocrity. No comfort from the pain zone. No pride to take home. But we are paying the bills, with more left over than most have. No broken backs and long hours. No lack of health care, no abuse from customers or the public. Not even that big a worry about job security. We are the lucky ones. We are blessed with a golden shovel. So let's do like those blue collar laborers we often idolize, and get to this annoying, bloody awful work that we are blessed with.

torginus• 2 years ago

I just wonder what would happen to society if everyone worked like software engineers do.

For example, what would happen if carpenters just eyeballed every measurement, and just shrugged their shoulders when the walls didn't line up. If they found out the way the wiring was planned will not actually end connecting where they thought it would and just shrugged and did it anyway.

What if they had hour long meetings about how to drive in screws and their workdays consisted of putting up a beam, and then going home because they think they've done enough for the day.

If all of society was ran like that, it wouldn't be running for long.

Lanolderen• 2 years ago

In all honesty construction is full of hack jobs and the buildings are standing.

icedchai4• 2 years ago

100% this. It’s still sad to experience this though, day after day. I’ve seen fundamental things done so badly, no time to fix them, we’re given no choice but to continue “construction” on top of a smoking garbage pile. Nobody wants to hear “start over.”

hermitcrab9• 2 years ago

That seems a rather defeatest attitude. I was fed up with working for other people 20 years ago. I started my own 1-man software company and write software how I think it should be written. The software isn't perfect, but I'm proud of it. I'm not rich, but I do fine financially. It isn't a viable path for everyone, but it is something to consider if you hate your job.

0xbadcafebee2• 2 years ago

I am defeated. I think a lot of us are. It's wonderful that you have your own company, but I don't have the grit and self-discipline for it, and I'm still trying to scrape up enough for retirement. A younger me might've liked to try your path, if I'd heard more of those success stories growing up.

ludicity21• 2 years ago

I'm the author. This is the track we're on. We also named our company "Hermit" so there's something extremely serendipitous about this comment, hah.

I definitely think it isn't viable for many people, but I've also met people who could totally do it that are scared because they're around a lot of people with ability/financial constraints that they simply don't have.

int_19h• 2 years ago

The other aspect of this is that it's easier to be the cog when you're tasked merely with implementing shitty decisions. You can just disregard the rationale and focus on the list of things you're told to do.

But as you (hopefully) progress in your career, the expectation is that you are increasingly the one making and justifying those lists, which is much more soul crushing and harder to "background" and pretend that it doesn't matter.

nyarlathotep_2• 2 years ago

> I've worked for like a dozen companies full-time. Most people don't know what they're doing.

This is why I reflexively chafe when I see "Software|Data Engineering".

I've never witnessed it in practice anywhere I've worked.

Sure, embedded systems for a pacemaker/defense industry or something is probably proper engineering, or graphics programming where you're concerned with creating approximations of real light transport with a time budget per frame or whatever, but most of this over-architected junk involving putting strings in databases and sending clients 10MB of JS to render a list is the farthest thing from "engineering".

A decade+ of inflated compensation really went to people's heads with these silly titles.

tofflos• 2 years ago

Like why didn't anyone catch the issue with the logs?

I see questions like these a lot and every time I feel that people immensely underestimate the effort required for curating data. In my experience data can only ever be as good as what it's being used for and in this story the logs haven't been used for this purpose before so they're not going to be any good.

It's some sort of data variation on the second law of thermodynamics - entropy is winning. Going in with the expectation that things should be better will only lead to frustration.

epgui• 2 years ago

This is not a data curation issue though, it’s a basic o11y issue.

andrewflnr• 2 years ago

o11y

Seems to be "observability", for anyone else seeing that for the first time.

t420mom• 2 years ago

I feel like there's a whole new generation of tech workers that need to read _Zen and the art of motorcycle maintenance_

ludicity21• 2 years ago

Author here. This is one of my favourite books of all time. In fact, my two favourite books are ZAMM and The Black Swan, both of which I hated on my first read when I was 19.

I recently re-read it for the third time while on holiday and was taking notes on pages worth including in a book review. I ended up with about 1/3rd of all pages logged and decided no review was better than just telling people to read the book, but I might write a blog post on gumption traps.

nathancspencer• 2 years ago

of course, we're serverless, because how can you hurt yourself without a cutting-edge?

Brilliant

zbyforgotp• 2 years ago

Bullshit jobs once again. I don’t know. These companies are complex systems.

He is writing as if the engineers all knew how to fix the systems, but were just powerless to do that. But I’ve also seen projects lead by engineers that only added to the overall complexity.

There is a paradox in this - the people who seem the most confident about fixing the systems usually only make things worse. Chesterton fences and stuff.

This article triggers me because everybody who reads it will always believe that they would fix the mess if only they got the power, but in practice when they get power they would only add new complexity to the whole mess.

xena• 2 years ago

People generally do have some idea how to fix complicated systems have have endemic problems. The reason they don't is that the company considers the capitalism going up faster NOW to be way more important than it going up faster in the future.

mrlonglong• 2 years ago

This makes me very glad I now write software that drives hardware.

paulsutter• 2 years ago

Understanding that IT projects are difficult gives us more empathy. Gartner says that 80% of corporate IT projects are considered failures. McKinsey says that 17% of large projects fail so badly that the companies existence is threatened. Standish group says only 10% of projects succeed.

hinkley16• 2 years ago

All large successful projects start as small successful projects.

sethammons• 2 years ago

Like why didn't anyone catch the issue with the logs?

Because there were no automated tests. If the company needs something to work, that thing needs a, preferably automated, test.

pards• 2 years ago

The word enterprise means that we do this in a way that makes people say "Dear God, why would anyone ever design it that way?"

Thank you for this phrase; I'll quote it at every opportunity.

hinkley16• 2 years ago

I’m working on an SDLC app that will end up with inspirational sayings as interstitials once I run out of bigger features/desperately want to procrastinate.

I’ll stick heavy hitters from Goldratt, Fowler, Feynman et al in there, but there’s going to be a “dark humor” and “snarky” category and this will definitely go into one of them, along with some Ambrose Bierce.

tokinonagare• 2 years ago

At two of the four businesses I've worked at, the most highly-performing engineers have resorted to something that I think of as Pain Zone navigation. It's the practice of never working unless pair programming [...] The fear and dread comes from a culture where people feel bad that they can't work quickly enough in the terrible codebase

Exactly why I burned-out at work, worked at most 2 hours per day on a good day and finally was ejected from the project after a PM that graduated last year from school noticed and went after my head. Author is a wizard for describing the situation this well.

It's been 3 days I've been free from the tyranny of Jira and project managers, and I worked more on my personal projets than I did in a week at my former workplace.

hinkley16• 2 years ago

The word enterprise means that we do this in a way that makes people say "Dear God, why would anyone ever design it that way?"

I feel this comment in my bones.

schnatterer• 2 years ago

My search for a generic definition of the term enterprise finally comes to an end :D

kayo_20211030• 2 years ago

Good Grief! Prolix?

coated in grass which rends those who tread upon it like a legion of upraised spears,

and there's more. Get to the point, whatever it is.

dcminter• 2 years ago

Some of us enjoy the journey.

hinkley16• 2 years ago

This is the “I saved my company half a million dollars in about five minutes” person who got in trouble for saving them half a million dollars.

ludicity21• 2 years ago

You are the only person in months that has called me anything other than "the AI rant guy" and I just wanted to say I appreciate you. <3

hinkley16• 2 years ago

I am an AI agnostic so even if I remembered that was you too, I’d only say it to our tribe.

I came into the industry exactly as a previous AI Trough of Disillusionment was in full swing so I’ve been vaccinated.

sampo• 2 years ago

The "AI rant": https://news.ycombinator.com/item?id=40725329

mildzebrataste• 2 years ago

This snippet is so relatable, so great:

"…what they actually needed to do was fire most of the staff in every team, leaving behind the two people who actually had good domain knowledge, then allow them to collaborate with good engineering teams to build sensible processes and systems.

Instead, they hired a bunch of Big Firm Consultants. You can see where this is going already."

I’m on month ten of my tech sabbatical and it’s been great. I’m no closer to wanting to return to the industry (former FAANG data jockey), not in a position where I can never go back but am in that middle-state where I want to use tech to contribute to the betterment of society, the community, nature and sustainability. I have a couple more months to try and figure out how.

sourcepluck4• 2 years ago

I had read and very much enjoyed the "AI silence or you'll be Piledriven into next week" post, without clicking on anything else on the author's blog. It was click link, read whole thing, love it, send it to one or two people, move on.

Very happy to see this here, realise it's the same person and that this is "a thing", and then to rollick in the author's backlog. A joy! Raucous real-life laughter has exploded from me on numerous occasions along with most articles. I think I've read 5 in a row there, and my brain is buzzing happily.

Thank you to the author for having the courage to write about real experiences. A breath of fresh air. I look forward to future books and articles, and reading more previous work, and cross my proverbial fingers hoping they can keep it real in the face of what will presumably be an avalanche of grifters looking to leech off the attention.

refulgentis3• 2 years ago

for fellow readers: https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...

sourcepluck4• 2 years ago

And, adjacent, in my opinion, readings like this piece, entitled "Hallucinating sense in the era of infinity-content":

https://www.documentjournal.com/2024/05/technical-images-fil...

hermitcrab9• 2 years ago

I did suggest this excellent link to a well known data science newletter, but they said it was a bit too profane for their readership!

jacobyoder• 2 years ago

"Suffice it to say that while people are sincerely trying their best, our leaders are not even remotely equipped to handle the volume of people just outright lying to them about IT."

I've tried to come up with some heuristic to determine whether or not a team is competent, good, or doomed. I've been exposed to all over the last... 8-10 years, and one of the key things I've noticed is the ratio of competent/skilled developers to the unskilled ones is a big ... indicator(?). Predictor?

Colleague of mine has been working with a team - dev team has ranged from 5-8 people over the last few years. Few people seem to have any grasp of programming at all. Only two people - my colleague and one other - have ever taken projects from ideas to delivery, or even taken features from requests to successful rollout of already functioning software.

The arguments that people get in to there - days or weeks of people 'researching' whether or not OAUTH 'really' requires 'refresh tokens' or whether it's really supposed to be a JWT. Management has some notion of 'every voice is legitimate and should be heard - we don't support bullying' and so on.

If you have a team of 10, and 1 or 2 people are simply bad at having the ability to think somewhat abstractly, you can survive.

If that number hits, say, 4-5... the team will struggle. A lot. You can keep things going, but it will be slow. And everything becomes a battle.

If that number becomes 7 or 8, and you only really have 1-2 developers who are actually competent developers... things will continue to spiral downward.

On the other side - I worked with a team of about 8-10 people on a 6 month contract. The larger org had another 40 or so folks, handling other projects, and support. Onboarding was great - I pushed production code in the first week. Everyone on the team was competent, including the juniors. I had more development experience, but they had more company experience, and it was really a relatively enjoyable engagement overall.

It was refreshing to be able to ask anyone on the team questions, and either get a workable answer, or an "I'm not sure, let's check with XYZ" to get working answers. The "oh, yeah, it's ABC" when ABC is clearly not the answer stuff never happened. People committing code and pushing to production without ever having run the code at all - I've experienced that - didn't happen - that's happening to my colleague.

The problem with a plurality of tech-incompetent folks in a tech group is that they honestly can not determine that they aren't competent. The only examples of competence are in the minority, and tend to not be trusted (even though that minority is the only portion that turns out working/functional code).

Leaving ends up being the only option in those cases. My colleague is only at his place part time, and has hung around because they've gone through some restructuring where new folks were brought in, and... you hope that things might get better in a few months, then realize they don't.

javajosh• 2 years ago

Tempting to write a response piece, "Get me into data hell, I need the money."

ludicity21• 2 years ago

In Australia, data jobs have a high floor but a low-ish ceiling at most places. Salaries start at around A$100K if you know how to interview, but 30 years of diligent practice only nets you around A$200K, and this would be considered an extreme success.

If you're already good, just have coffee with executives/leads at places that take engineering seriously and you'll earn well, but there's no need to enter data hell as an employee at the typical company.

pugz• 2 years ago

I'm also based in Melbourne. I would add that all the above is true of maybe 95% of companies, but there are a few exceptions: big US-based tech companies. There are roles for devs that pay AU$250-$550K/year, including the data engineers. Unfortunately, there appears to be zero correlation with expertise and we have the exact same pain zone experience that I had when I was earning a third of this salary at an AU-headquartered company.

ramshanker• 2 years ago

Acting against your own better Judgement, I have to take that fight/ urge every few days. ;)

anitil• 2 years ago

I really enjoy this writing for a couple of reasons. Firstly, Australian. Secondly the acerbic wit just tickles my fancy

we're serverless, because how can you hurt yourself without a cutting-edge?

Just perfect.

Get me out of data hell

Related Stories