Engineers Shouldn’t Write ETL

http://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl

by mjohn • 10 years ago

291 174 10 years ago

multithreaded.stitchfix.com

This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.

tantalor• 10 years ago

Agreed. I think ETL = https://en.wikipedia.org/wiki/Extract,_transform,_load

kafkaesq5• 10 years ago

It's a pretty common acronym in the field, actually.

toufka• 10 years ago

There are many of us here who are not in the field.

justinlardinois• 10 years ago

Maybe in the data science field, but not in the tech field more generally. Probably doesn't help that the thread title didn't specify it was data science-related.

mikestew3• 10 years ago

Though I agree with you on expanding TLIs, if you have to have "ETL" defined for you, you probably won't get the "joke". And though this will come out more cynical than I intend, if you don't know the acronyms, then you probably won't be buying what Stitch Fix is selling. Filtering their funnel, maybe?

ams61102• 10 years ago

While possibly true, it's simply a courtesy to the reader to parenthetically define any acronym the first time it's used in a published piece of writing (of course this would not apply to internal emails, casual comments such as discussion forums here, etc.)

ksenzee• 10 years ago

Hm? Stitch Fix is selling clothes, not software. This is just their engineering blog.

kaonashi• 10 years ago

Might end up sounding like this: https://www.youtube.com/watch?v=BpMwZDfSLBk

arethuza• 10 years ago

ETL, DBA, API - aren't they all incredibly standard acronyms?

skj• 10 years ago

Not to me. I know what DBA and API are, but I've got no idea what ETL is.

kelvin02• 10 years ago

Standard for you means that we live under rocks? Obviously if it's standard we must be clueless. Nice way of phrasing it.

yxhuvud• 10 years ago

TLA is the common abbreviation for what you call TLI, that is Three Letter Abbreviation.

The common abbreviation for a four letter abbreviation is ETLA. That is, Extended Three Letter Abbreviation.

TeMPOraL2• 10 years ago

100% agreed. I was clueless about ETL too. While the acronym itself may be old, people forget that this field is hip and sexy for maybe like last 3 years or so.

hodwik• 10 years ago

Nope.

https://www.google.com/trends/explore#q=extract%20transform%...

hnal9432• 10 years ago

Google fixes your ignorance instantly. This TLI is common enough to assume.

joeblau• 10 years ago

I'm getting a DNS error - could be me though.

mikestew3• 10 years ago

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”.

OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh, wait, I need Cassandra on my resume, so Mongo sux0r now...displayed with some of the worst bowl of spaghetti Android code I've witnessed. The technical debt hole was dug so deep you could hide an Abrams tank in there. To this day I could not tell you the confused thinking that led them to believe this was necessary rather than just slapping it all into PostGres and calling it a day.

All because they were processing data sets sooooo huge, that they would fit on my laptop.

I quit reading about the time the article turned into a pitch for Stitch Fix, but leading up to that point it made a good case for what happens when companies think they have "big data" when they really don't. In summary, either a company hires skills they don't really need and the hires end up bored, or you hire mediocre people that make the convoluted mess I worked with.

jboggan2• 10 years ago

I thought it nailed a lot of dynamics at my last gig as well. Of course, I joined for the challenges of scaling and wrote an ETL framework in Rails (that was a mixed bag but very instructive) and then got bored after I realized how small our data really was. Then I left for Google and all my data went up by two prefixes.

I do love reading an article that supports my contention that ETL is the Charlie Work [0] of software engineering.

0 - http://www.avclub.com/tvclub/its-always-sunny-philadelphia-c...

kafkaesq5• 10 years ago

It's all Charlie Work for the VCs, "founders", and (occasional) early hires who take in the lion's share of profits and prestige for the long hours most of us pour into our jobs -- day after day, year after year.

BTW, that may sound polemic, but it's not -- that's really how a lot of business types, academic researchers, and others think of nearly all programming-related work.

mikestew3• 10 years ago

I do love reading an article that supports my contention that ETL is the Charlie Work [0] of software engineering.

Though I've never formally done work with the "ETL" label, what I've seen of it reminds me of the work I used to do for client years ago where I'd take some CSV file (or what have you) and turn it into something that their FoxBase-based accounting system could use. It was boring grunt work (or shall we say, "Charlie work"), but I billed it at my usual rate and it paid the mortgage. I would never, ever wish to make a career of it, however. (And if my assessment of what someone knee-deep in ETL does all day is completely off base, I apologize.)

tbrock• 10 years ago

Like all things in tech, it depends. I agree with you that most startups will never see data that will not fit in memory. Remember http://yourdatafitsinram.com ?

Now, a data warehouse where you have a hoarde of Hadoop engineers slaving over 1000 node clusters is probably overkill unless you are a company like facebook or are in the data processing biz. However, some database management systems can be very complementary.

For example, while you can do a majority of things with Postgres, some are not fun or easy to do.

Have you ever set up postgis? I'd much rather make a geo index in MongoDB, do the geo queries I need and call it a day than spend a day setting up postgis.

We find that MongoDB is great for providing the data our application needs at runtime. That being said, it's not as great when it comes time for us analyze it. We want SQL and all the business intelligence adapters that come with it.

Yeah you could do a join with the aggregation framework in MongoDB 3.2 but it just isn't as fun.

darksaints2• 10 years ago

Have you ever set up postgis? I'd much rather make a geo index in MongoDB, do the geo queries I need and call it a day than spend a day setting up postgis.

I think you should try it again. I felt the same way when it was in the 1.5 version, but post 2.0 it is really easy. And the documentation has gotten so much better. It really is as simple as `sudo apt-get install postgis` and then `CREATE EXTENSION postgis;" in psql.

darksaints2• 10 years ago

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, service logs), ETL can get really absurdly complicated. But that is still no excuse to introduce big data if your data isn't actually big.

I really hate pat-myself-on-the-back stories, but I'm really proud of this moment, so I'm gonna share. One time a principal engineer came to me with a data analysis request and told me that the data would be available to me soon, only to come to me an hour later with the bad news that the data was 2 terabytes and I'd probably have to spin up an EMR cluster. I borrowed a spinning disk USB drive, loaded all the data into a SQLite database, and had his analysis done before he could even set up a cluster with Spark. The proud moment comes when he tells his boss that we already had the analysis done despite his warning that it might take a few days because "big data". It was then that I got to tell him about this phenomenal new technology called SQLite and he set up a seminar where I got to teach big data engineers how to use it :)

P.S. If you do any of this sort of large dataset analysis in SQLite, upgrade to the latest version with every release, even if it means you have to `make; make install;` Seemingly every new release since about 3.8.0 has given me usable new features and noticeable query optimizations that are relevant for large query data analysis.

RivieraKid• 10 years ago

What technologies / architecture does Amazon use for business intelligence? I've just done a business intelligence course so I'm interested how do more "technology-centered" companies approach BI, whether it's the same thing I learned in the course (put everything into an integrated relational database if I simplify a lot).

ignoramous3• 10 years ago

Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention eliminating cost and other head-aches; kind of expected since rodb is an embedded db...

lomnakkus2• 10 years ago

The really fun aspect of this for me, personally, is that I've been doing computers for ages (25 yrs or so?) now and 3TiB still intuitively feels like a massive amount of data even though I think I have something like 6TiB free space on my home server disks... which, in total, didn't even cost as much as a months' grocery shopping. Sometimes it really takes effort to rid yourself of these old intuitions that don't really work any more.

(I'm getting better at it!)

EDIT: And my desktop machine has -- let's see -- about 500000 times the RAM my first computer had. Truly astounding if you think about it.

chubot• 10 years ago

That's awesome... I'm working with some 15-20 GB sqlite databases. Though 2 TB sounds kind of big?

Was it 2 TB before compression? Because sqlite does blow up the data size over the raw data usually (depending on the original format obviously). It can be kind of wasteful, and I ended up storing some fields as compressed JSON for this reason (that actually beats the sqlite format).

Also, the sqlite insert speed can be much slower than the disk's sequential write speed (even if you make sure you're not committing/flushing on every row, and if you have no indices to update, etc.)

So I think inserting and laying on the data could be nontrivial. But the queries should be fast as long as they are indexed. In theory, sqlite queries should be slower for a lot of use cases because it is row-oriented, but in practice distributed systems usually add 10x overhead themselves anyway...

Buttons840• 10 years ago

Me and a coworker were laughing at the parent comment, and I told him:

"I guarantee that somewhere, sometime, an engineer has been like 'hay guys, I loaded our big data into SQLite on my laptop and it ended up being faster than our fancy cluster'". We then joked that the engineer would be fired a few weeks later for not being a "cultural fit".

A few minutes later you commented with your story. I hope you didn't get fired? :)

banku_brougham• 10 years ago

thanks for the SQLite tip, I've been meaning to add it to my tool set.

question: is SQLite incrementally helpful when I'm already comfortable with a local pgsql db to handle the use case you suggested? would SQLite be redundant for me in this case?

question: between postgres and unix tools (sed, awk) is there reason to use SQLite?

pacala• 10 years ago

In all fairness, one can run Spark on a single machine. The key insight and that "single machine" has gotten pretty big these days, the bit shuffling technology may be secondary.

ap222132• 10 years ago

How long does it take you to load 2 TB into SQLite? How long do queries take? I believe you, but I'm in disbelief that it could be close to as efficient as throwing into ram. I mean, an EMR cluster takes like 5 minutes to spin up.

Where do I learn how to do this? I've tried loading a TiB (one table one index) into SQLite on disk before, and it took forever. Granted this was a couple years ago, but I must be doing something fundamentally wrong.

I want to try this out. I've got 6TiB here of uncompressed CSV, 32 GiB ram. Is this something I could start tonight and complete a few queries before bed?

Actually, out of curiosity, I looked it up on the sqlite site. If I'm reading the docs correctly, with atomic sync turned off, I should expect 50,000 inserts per second. So, with my data set of 50B rows, I should expect to have it all loaded in ... Just 13 days. What am I missing?

Trundle• 10 years ago

Hey, would you be willing to go a little in to your background/qualifications as well as maybe what your daily/weekly workload and goals look like? BI is a field I'm curious about because it sounds very interesting, very important, and as far as I can tell, with tons of surplus demand even for entry level. It makes me wonder if the actual day to day must be horrible for those things to be true.

x0x0• 10 years ago

I worked at a place that shall remain anonymous. An engineer later terminated for cause spent almost 2 months setting up a hadoop cluster with all the stuff that goes along with it to analyze some log data. When a senior engineer manager finally got incredibly tired of waiting for the results, said senior engineer manager wrote the desired analysis in an afternoon in ruby, and it ran in a couple hours mostly limited by the spindle speed on his laptop drive.

The productivity factor of ruby or python in a single memory space vs hadoop is at least 10x.

busterarm• 10 years ago

People frequently think that their sub-petabyte dataset is a big data problem.

Time and again, the tools prove that it's not.

To recollection, I can count the number of publicly-known companies dealing with these datasets on two hands, if being generous.

emcq2• 10 years ago

My favorite is when you jump on a project and do a simple estimation of compute throughput for the highly complex distributed system, its something like hundreds of kilobytes per second. You could literally copy all the files to one computer and process faster than the web of broken parts. It becomes cancerous; to work around system slowness ever complex caching mechanisms and adhoc work is constructed.

I think part of the problem is many engineers can debug an application, but surprisingly few learn performance optimization and finding bottlenecks. This leads to an ignorance is bliss mindset where engineers assume they are doing things in reasonably performant ways and so the next step must be to scale, without even a simple estimate for throughput. It turns into bad software architecture and code debt that will cause high maintenance costs.

lovich• 10 years ago

Would you happen to know a good way to learn performance optimization? I'm working with datasets currently that I am trying to get to run faster, and I cannot tell if the limitation is on my hardware or due to ignorance on my part

logicallee• 10 years ago

data sets sooooo huge, that they would fit on my laptop

well yes this is an embarrassing fact for data scientists. A midrange stock macbook can easily handle a database of everyone on earth. In RAM. While you play a game without any dropped frames.

gedrap• 10 years ago

This is very very true, very often when it comes to more than a gigabyte of data or more then a hundred of queries a second. Quite a few inexperienced devs suddenly think it's big data they've been reading about and FINALLY they can play with it!

With all the hype around 'big data' and all that crap, many people seem to forget how far you can go with plain simple SQL when it's properly configured, and not talking about some complicated optimizations, just solid fundamentals. And no problem if you can't do it, things like Amazon RDS will help you.

im_down_w_otp• 10 years ago

Indeed. Whenever I give a talk about "big data" I make sure to include the phrase, "I'm pretty sure most of the big data market exists on the hubris that developers want to desperately believe their problems are bigger than they really are."

romming• 10 years ago

I think there's a more common reason why companies end up with "awful-to-work-with messes": ETL is deceptively simple.

Moving data from A to B and applying some transformations on the way through seems like a straightforward engineering task. However, creating a system that is fault-tolerant, handles data source changes, surfaces errors in a meaningful way, requires little maintenance, etc. is hard. Getting to a level of abstraction where data scientists can build on top of it in a way that doesn't require development skills is harder.

I don't think most data engineers are mediocre or find their job boring. The expectation from management is that ETL doesn't require significant effort is unrealistic, and leads to a technology gap between developers and scientists that tends to be filled with ad-hoc scripting and poor processes.

Disclosure: I'm the founder of Etleap[1], where we're creating tools to make ETL better for data teams.

[1] http://etleap.com/

vikiomega92• 10 years ago

It depends a lot on what you're doing, but 2 TB in general seems a lot. If you have to perform an out-of-disk sort, you probably need a distributed setup. The funny thing is that most setups that use EMR defeat the data locality principle and I think that's the speedup people experience when they run it on a single laptop for example. Reading the original Google paper helped me a lot in understanding this.

rspeer3• 10 years ago

If you have to perform an out-of-disk sort, buy another hard disk and now it's not out-of-disk anymore. This will suffice for nearly any data set you would ever need to sort.

A 4 TB drive costs about $120, and you'll spend way more than that on software development and extra computers if you do distributed computing when you don't need to.

zeroecco• 10 years ago

This article is so one sided it is painful. it is almost like Sheldon Cooper wrote this article. As an engineer I am offended and hurt that we are referred to as “Tony Stark’s tailor”.

ellimilial• 10 years ago

Btw didn't he build the suits himself and just share it with his less able cop friend?

I'd agree, it feels like it invalidates the claim about getting credit for 'being a thinker'.

MrFoof• 10 years ago

You Probably Don’t Have Big Data

"Big Data" is like sex in high school. Everyone talks about it but few people really have lots of it and some just don't have any.

kafkaesq5• 10 years ago

And among those that do, fewer still know how to do it particularly well.

strictfp• 10 years ago

The thing is that everybody has big data, it's only a question of how much of your data you save.

tyre2• 10 years ago

This is not true.

For many startups, even if they audit everything they won't have petabytes of data.

0xdeadbeefbabe2• 10 years ago

"Big Data" is like good analogies. Few people really have lots and some just don't have any. ;)

kod• 10 years ago

The idea that engineers should build lego blocks without knowing what they're going to be used for is questionable at best.

A better idea imho is to have small crossfunctional teams where scientists and engineers work together to build only what they need with short iteration cycles.

If everyone involved doesn't have at least a broad perspective on the end-to-end purpose of what they're working on, they're probably going to build the wrong thing.

0xdeadbeefbabe2• 10 years ago

Although, what you say about lego blocks applies to iterations, which are lego blocks in time.

kafkaesq5• 10 years ago

There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume.

I'm not sure I get why writing ETL code for data you'll never consume is any more soul-sucking than, say, refactoring JS code for a website you couldn't begin to care about (and which will never be properly re-designed anyway); or even doing "thinker"-level work but for an industry you couldn't begin to care about (advertising), etc.

In other words, what most developers of whatever technical stripe do for a living.

kaspm• 10 years ago

And I also fundamentally disagree with the notion that moving a large amount of realtime data reliably and with accuracy, monitored and consistent with relatively little failure is not an interesting engineering challenge in itself. I find that for all the talk about data-driven organizations, most don't use a tenth of what is available but that when the tenth is needed, it's hugely satisfying to be able to provide it.

kafkaesq5• 10 years ago

I'm with you. I hear people reflexively dissing ETL (and other aspects of front-line data engineering) all the time, but I've come to suspect they don't really know these systems are actually about.

dspillett• 10 years ago

And I also fundamentally disagree with the notion that [ETL work] is not an interesting engineering challenge in itself.

A lot of people think that certain DBA/ETL/BI/similar work is boring and simlpy don't want to do it and so don't learn to do it well. Which is fine by me: it means those of us who can do it well can get paid good money when someone needs it.

The only problem with this theory in practise is that many also think such work is easy and free of complications; so they baulk at paying for people genuinely can do it well, get people less experienced who say they can do it well but do it badly, and judge the rest of us by that standard and assume database people are thick and can't do easy jobs properly...

hakann• 10 years ago

I work at a small startup with only 2 people in analytics. We build the infrastructure, data pipelines and do the BI analysis and data science. And I really enjoy knowing how it all comes together and being able to change anything in the pipeline. Maybe not having enough money for a big data department is our blessing.

lordnacho• 10 years ago

He hits upon a quite interesting division of labor. Where I've worked in finance, there's been "strategists" and there's been "developers". You can guess which one is seen as high prestige.

The problem arises when someone gets into a position where they can think big thoughts without having to do any nitty gritty. Effectively, they end up jumping in right when the real producers have finished the actual work, and then coming up with some polish that makes it look like they came up with some interesting result.

This is not actually a way to get work done. It's a way to play politics.

Worse yet, it's actually completely detrimental to getting things done. When you have things split up between thinkers and doers, what do the incentives look like? It's quite simple. I may order some analysis, and I may not fully understand the nuances. But whatever happens, as a thinker I'll have to have something grandiose to say, and I'll need to keep the doers busy. That way if I don't find a real conclusion, it's everyone's fault. If I do find something, it's thanks to me.

Where I worked the people with the big plans couldn't code their way out of a paper bag. Ask them what Big-O is, they draw a blank. Ask them how their trading strategy will actually send orders to the exchange, they draw a blank. But ask them something that sounds like strategy, and they will feed you plenty of unsubstantiated BS.

My new venture is coders all the way down. Strategists who can actually use git without asking what it is, understand that algorithmic complexity actually matters, and so on. Coders who understand what the market is.

BenoitP• 10 years ago

The more I think about making credit match with work in an organisation, the more it looks like a neural network where the credit/revenue/profit gradient is having problems being back-propagated.

In the horizontal setup you describe (layer of thinkers on top of layer of doers), credit hits a barrier at the thinkers. The gradient isn't propagated.

In the vertical setup in the article (layer of thinker-doers), of course the backprop will be good because it is only one layer thick. You gain proper incentives, proper treatment of data on the whole pipeline. And the engineers can also concentrate on a purely orthogonal thing: writing tools.

But you lose the benefit of having the layer being able to focus on one thing. The author acknowledges those efficiencies (his word). It is hard to find people with a wide set of skills. Although in this case it is balanced, as now the engineers have gained specialization.

But I digress. My point was: humans in orgs are bad at backprop. Why share the credit at all? Organisations can be seen as neural networks/graphs, and they can lack proper backprop.

I'd love to see the results of some pagerank-like backprop. Every employee gets one base point. Every week, he is asked: "who helped you the most in doing your job this week?". Sales would credit analysts who would credit engineers, etc. Or Sales would credit analysts-engineers who would credit tool-writers, etc. It could go both ways: engineers could credit sales or analysts for writings well thought-out problem descriptions.

Then you would run pagerank on it, and base every promotion, every salary increase on it. Information would flow well, and everybody has a clear direction (his gradient) of what he can do to shine.

Also, by injecting revenue at the sales layer in a certain period of time, you could identify who conctributed the most in an increase of revenue.

Also, I posit that managers have a tiny view of what happens in a firm. They only get to see a fraction of interactions, while the brunt of what matters happens in the long tail of one-to-one interactions. Should you chose to promote people with the highest PR, you would have a true result-based bottom-up org.

ZenoArrow• 10 years ago

“What is the relationship like between your team and the data scientists?” This is, without a doubt, the question I’m most frequently asked when conducting interviews for data platform engineers. It’s a fine question – one that, given the state of engineering jobs in the data space, is essential to ask as part of doing due diligence in evaluating new opportunities. I’m always happy to answer. But I wish I didn’t have to, because this a question that is motivated by skepticism and fear."
"Rather than try to emulate the structure of well-known companies (who made the transition from BI to DS), we need to innovate and evolve the model! No more trying to design faster horses…

A couple years ago, I moved to Stitch Fix for just that very reason. At Stitch Fix, we strive to be Best in the World at the algorithms and analytics we produce. We strive to lead the business with our output rather than to inform it."

I find this article rather peculiar. At the start, you'd be forgiven for thinking this was an article about a company looking to find a solution to a problem, but as the article progresses it's clearer that they're selling themselves as the solution to the problem they outlined.

In other words, they start off looking like a customer, but only to set up the premise required to sell the solution to the problem their company supposedly has/had. Turned me off from taking the product seriously.

lafay• 10 years ago

Is ETL really even necessary anymore? Why not just run fast ad hoc queries over the raw data with something like Google BigQuery?

web007• 10 years ago

Yeah - just Extract it from your MySQL / Mongo / Postgres / logfiles / whatever system it's in right now, Transform it into a CSV or whatever the input needs to be and Load it into BigQuery. Once it's there, you can do whatever you need!

ams61102• 10 years ago

On a smaller scale, the "q" utility has been a boon for me in the handling of ad-hoc delimited data files.

http://harelba.github.io/q/

Really one of the best things I've discovered in the past 5 years. Saves so much work compared to doing stuff with sed, awk, and the like.

v645• 10 years ago

Getting the raw data into BigQuery or another tool is an ETL problem in itself. You have unstructured log files, you have external APIs such as SalesForce, you have various relational databases, etc etc. Someone has to come in and transform this data into a unified format that can be inserted into BigQuery.

hrvbr• 10 years ago

Cheers for the beautiful website that doesn't require javascript.

phamilton• 10 years ago

We are not optimizing the organization for efficiency, we are optimizing for autonomy.

This is one of the toughest parts of building a scalable organization (with or without big data). Getting past the idea of efficiency and being OK with redundancy.

This means allowing two teams to both build a common feature they might need, rather than establishing a dependency. It means making one teams job broader even if it overlaps with another team.

I find it interesting that we are perfectly willing to have redundancy on the software side (load balancing, slaves, etc) but not on the development side.

dizzystar• 10 years ago

Nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise then that ETL engineering roles are the archetypal breeding ground of mediocrity.

There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume.

This is like... your opinion. Some people find pushing around HTML / JS / CSS absolutely soul-crushing. Considering the lion's share of websites are ugly, unusable, and slow, does this mean that front-end engineering is a breeding ground of mediocrity, so server-side devs and CFOs should all be sharing in the pain?

Some people actually enjoy working with data, and don't find ETL and pipelining horrible to do at all. It is a different set of challenges, but calling people people mediocre because of ETL is a non-sequitur.

demic• 10 years ago

this.

bazqux2• 10 years ago

I love ETL. I've worked with real Big Data (5PB+) and working on the data pipeline was my favorite part. The feeling you get when you rewrite a job to run 1000x faster so the company can make way more money.

Terr_• 10 years ago

That implies the work is in a revenue-center or that you're at the rare company that isn't myopically focused on sales.

shubacca• 10 years ago

I don't think the author would disagree with you on that. His main point is that engineers don't want to be in a role where all he/she is doing is productionizing someone else's ideas. There should be areas of end-to-end ownership for both the engineering team and the data science team.

It doesn't sound like he's suggesting here that some types of technology are just generally horrible to work with or that it sucks to build ETL jobs in general. Maybe he could have done a better job defining what he meant by ETL engineer here, but he does qualify it in that quote with "ETL to produce data that you yourself never get to use or consume."

In general, roles that offer little areas of ownership do draw mediocre engineers.

MechanicalTwerk• 10 years ago

Could not agree more. With all this talk of data and ideas, you would think he could include some data to back up his ideas. He can't, because this is all just opinion.

fizixer• 10 years ago

Data-driven decision-making to change the course of a business, is so internally disruptive it's unlikely to happen in an org-chart culture full of management layer.

Because that's what it is:

- It is attempting to question, critique, override, everyday decisions made by the management (including the CEO) based on available data.

- It is doing that with maximal knowledge of the whole organization. That means all the records, finances, secrets, what not, have to be divulged to the data-science team. (which in itself is an unsurmountable challenge, i.e., to convince the management to allow full data access; think emails, chat logs, meetings minutes of CEO's, VP's, etc, etc).

This will make the management go, "so let me get this straight, I authorize you access to data of the whole organization, and you come up with a conclusion (some of the times at least) that I'm full of it?"

I highly doubt any organization would be up for this kind of internal disruption, even if that means more success for the company.

TheLogothete3• 10 years ago

Yeah, I definitely need the CEO's emails with cat pictures for my predictive maintenance models.

martini159• 10 years ago

Your comment so perfectly summarizes what I've long felt is the deep dark dirty secret of data science (at least at a company thats not Google/Facebook/etc). And it flies in the face of all the "top job" lists which are always littered with data-related jobs.

Very few people are interested in making data-driven decisions. They want an employee (subordinate) who will prove that the decision they've made or are planning to make, is correct. Anything else is, as you say, very internally disruptive.

Being a data scientist or data analyst at a startup is (for the most part) a completely miserable existence. You are relegated to doing interesting things that are usually discarded. It can make you feel like your job is pointless.

In the end, one either makes the decision to be (at best) useless, or (at worst) a puppet. That, or you quit.

Thank you so much for your comment - it's refreshing to see I'm not alone in feeling like this.

TeMPOraL2• 10 years ago

Someone with big enough clout should utter a proper rule of thumb at some prestigious software conference. Something like "if your data can fit on a single commercially-available hard drive, it's not big data". Maybe then it has a chance to filter down to university education over the next decade or so.

(Corollary to that rule of thumb: if your data fits on a hard drive, all "big data" tools you need are shell scrips and SQLite.)

ajnin• 10 years ago

This is not very convincing. He starts off by saying that the "traditional" model where the data scientists do the thinking while the engineers do the doing is unsuccessful because the engineers need to get invested in other people's ideas, need to maintain them and get blamed if they fail while the data scientists get all the praise. So he suggests replacing this with a new model where the engineers work horizontally aka in the shadows, have to be "Tony Stark tailors" and get out of the way while the data scientists get to be Tony Stark. Which is basically the same thing.

iblaine• 10 years ago

I think the conclusion is this. Data Scientists, Data Engineers, and Infrastructure Engineers exist in their respective roles. Data Engineers should enable Data Scientists to be better engineers by creating frameworks for Data Scientists. By doing so, Data Scientists will be less likely to put stress on everyone else.

Another point I'd like to make is that not everyone hates ETL and pipeline management. I happen to like it. It's rewarding to stand up reliable self-healing data pipelines and ETLs.

gtrubetskoy3• 10 years ago

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statisticians than engineers", etc. But then I disagree with the rest.

The distinction between "Data Scientists" and "Engineers" is bogus, and the point about whether your data is "Big" is a red herring.

In reality, there should not be any distinctions between "scientists" and "engineers", you must strive to be both a "doer" and a "thinker". You can't think without doing, and can't do without thinking.

If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking.

If you consider yourself a "data scientist" but don't know anything other than R and basic Python, go study programming and operating systems and networking, like you studied math at some point.

Somewhere on youtube I remember Dr. Donald Knuth (who is definitely an excellent programmer/engineer/computer scientist, arguably one of the best the world has known) saying that he considers himself primarily a mathematician.

Or, if you've read (or at least heard of) "the dragon book", you might find it interesting and inspiring that one of its main authors Dr. Jeffrey Ullman (whom I'd place in the same league as Knuth) went on to write another excellent (and available freely online, BTW) book "Mining of Massive Datasets", which IMHO is the one fundamental "big data" book out there.

So Data Scientists - go learn some programming languages like C and study UNIX and may be read "The Art of Computer Programming" and Engineers go read http://www.mmds.org/.

Then you'll all get along.

achompas2• 10 years ago

If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking.

This assumes availability of time. Obviously, given enough time, people could develop both top-tier engineering and DS skillsets!

Of course, if lots of free time were common we'd all be full-stack developers who also field sales calls and work on product strategy etc. etc.

gtrubetskoy3• 10 years ago

I don't know... Are you saying that someone like Donald Knuth had a lot of free time? Data problems are arguably the hardest problems out there and they require deep understanding of mathematics as well as computers, that's just the way it is. It does take time and effort and even may be a bit of talent as well, not everyone is cut out for it.

dxbydt• 10 years ago

This is a supremely ridiculous set of suggestions that has no merit whatsoever. Companies aren't libraries. They aren't paying you to sit and read books. There is an assigned dayjob, a set of tasks you have on your jira that you have to resolve by your deadlines, and that occupies the 8 hour workday ID you are doing any justice to it. So any reading you do is on the side, on your own time.

Furthermore, people have these roles precisely because of their talents and their choices. As a Data Scientist, most of what I do is read ML literature, build ML models and write technical reports in Tex on what worked and what didn't. The skills to do this were acquired over many painful years of graduate work in math, statistics, ML. To suggest somebody can just read their way through that material is quite laudable, but you are underestimating the difficulty by orders of magnitude. Essentially, you are suggesting that all of the graduate study and mentoring and homeworks and assignments and all that went into the learning process be condensed into a book which one can just plow through and become a DS. Well, good luck with that. By the same token, expecting me to have the same level of efficiency and passion as a data engineer when faced with a Hadoop/Oozie/Presto/Pig/kafka or what have you is silly. I don't care for these technologies and how to work them. I know it takes a really long time to get good at them - that's why the engineers get paid a lot of money and also get yelled at when the ETL job fails. Because it's a set of seriously valuable skills that were no doubt acquired over lots of time and practice. It's not like I can buy a book on these things, just read through them and suddenly I am a DE! I neither have the interest nor the time to do that.

>the distinction between data scientists and data engineers is bogus

Not at all! Both DS and DE professionals do distinctly different work and conflating everything under 1 umbrella buys you nothing.

gtrubetskoy3• 10 years ago

> I don't care for these technologies and how to work them.

And I'd argue that this is precisely the problem. I personally care about all of it.

leblancfg• 10 years ago

> Companies aren't libraries. They aren't paying you to sit and read books.

That stroke you've brushed is too wide. Smart employers will have some of the money they're paying an employee going towards learning... and if they're really smart, they can even measure their ROI. Leads to less turnover, and better long-term vision for their projects.

I get your point, but give someone passionate enough 6 months in a new work environment, and with a decent mentor, and you might find they become surprisingly adept at it. The hard part is hiring for the capability to learn (fast).

brown9-2• 10 years ago

If you manage to hire them, they will be bored. If they are bored, they will leave you for Google, Facebook, LinkedIn, Twitter, … – places where their expertise is actually needed. If they are not bored, chances are they are pretty mediocre.

Granted that yes, lots of solutions don't exactly require a Hadoop cluster with thousands of nodes, this is a pretty gross and mean-spirited dig at "mediocre engineers" a number of times. It would be nice if we didn't treat people that don't work at Amazon/Google/Twitter/LinkedIn as lesser beings because they find their jobs at a probably-doesn't-have-Big-Data company.

(Does StitchFix have Big Data? If the answer is no, are their "Data platform engineers" mediocre?

v645• 10 years ago

The fundamental flaw that prevents the Thinker and Doer model from living up to its recruiting hype is the assumption that there exists an army of soulless non-mediocre Doer engineers who eagerly implement the ideas and vision of data scientists.

There's a large, active community of engineers who specialize in data, whose job is to technologically enable data scientists the means to perform their analyses. I know these people exist because I'm one of them, and I work with them, and I've met them at meetups and conferences. I don't know why the author doesn't think these types of engineers exist. Not all of us who code want to work with the web.

If you read the recruiting propaganda of data science and algorithm development departments in the valley, you might be convinced that the relationship between data scientists and engineers is highly collaborative, organic, and creative. Just like peas and carrots.

Almost every data team I've worked with is structured this way. I work daily with data scientists. I have a data scientist sitting to my right, two data scientists sitting across from me. Our teams are highly integrated and I can't imagine it working any other way. If the teams the author is familiar with don't operate in this manner, then I can see why he'd think the endeavor is hopeless.

I also disagree with the author's conclusion. The data scientist's job is to analyze and interpret data. They should not be spending any time thinking about how to get that data. They should not be concerned about where the data is coming from. The more time scientists have to spend thinking about ETL, the less time they have to do what their training is in, statistical analysis.

nmkridler3• 10 years ago

I completely disagree, data scientists who can not create the data they need are at a significant disadvantage to those who can. Our job is more than being able to analyze and interpret data. If you have someone in your organization that spends no time thinking about how they get the data, you need to fire them or reduce their salary.

TheLogothete3• 10 years ago

I disagree with you.

v645• 10 years ago

The data scientists I work with are statistics PhDs. The extent of their programming knowledge is R and SQL. What are they supposed to do if the data they need to analyze is only available through a SOAP API you log into with OAuth, and they need to log in once a day to retrieve the latest day of data? Unless you're a software engineer, you probably don't have the skillset necessary to easily get that data.

The data we use comes from relational databases and document stores operated by different departments, external APIs and third party services, SalesForce, server log files, etc. A stats PhD does not have the training to gather this data themselves.

In terms of a hybrid scientist/engineer role, I don't know many software engineers who are also good at stochastic calculus or ensemble learning. Likewise, I don't know many data scientists who are also comfortable writing cronjobs to retrieve external API data or have the ability to diagnose server problems.

nissimk• 10 years ago

Title should have been "Engineers Shouldn't only write ETL." I agree with the author's statement of the problem, but not with the proposed solution. Succinctly, I think the problem is compartmentalization and specialization. These are qualities that are sometimes promoted by management so that it is easier to maintain control over the organization and to hire people who won't require much training to do their jobs. Unfortunately, compartmentalization and specialization both lead to unhappiness in the workers, and are net negative for production. I believe the solution is fostering a wholistic approach among the specialists. Data scientists (who should be statisticians or machine learning experts) should interact regularly with software development engineers that have to productionize their research and they should also both interact regularly with systems and database administrators who make it all work in production. Rather than being separate teams working on parts of the same goal, they should all be one team. By working together through the poroblems faced in each area, they can learn more about each other's areas of expertise and will create a better solution faster. This isn't true just for data science, but throughout technology, where operational software developers should work together with product development, marketing, testing and operations to break down the divide and get all team members working towards the same goal.

btilly3• 10 years ago

The thinker/doer problem goes way back. In most organizations the person who thinks of something gets the lion's share of the credit, and the person who implements it does the lion's share of the work. And if it turns out to be a bad idea, the thinker can always blame a bad implementation, thereby passing the lion's share of the blame to the doer.

I've seen careers made and broken based on whether people got to play thinker or doer.

This makes rewards for thinking very lopsided. However the problem is that actual credit for success REALLY belongs with the people who did the work.

This problem shows up at every scale in every organization. For example there are hundred people who want to be the business side of a startup for every person who wants to build the tech. Why? The business person gets to be the thinker, the developer does the work. And then the business person expects to become the CEO and get the bulk of the payout!

andrewflnr3• 10 years ago

But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.

btilly3• 10 years ago

And I didn't say that the doer always deserves the credit. As an example, who deserves more credit for the success of Apple, Steve Wozniak or Steve Jobs? Wozniak created the Apple I and most of the Apple II. But clearly Jobs' ideas built the current company.

However these cases are the exception, not the rule. As a rule ideas are cheap, implementations are hard. And success has more to do with iterating on the implementation than the starting ideas.

cookiecaper3• 10 years ago

"Ideas" alone are almost never worth anything. You have to do the work to back it up. Everyone I know has about a dozen ideas (you hear them all the time as someone who makes ideas real).

What matters is the technical skill to make the idea go from a fantasy to a reality semi-reminiscent of the idealized fantastic version, whether that skill is in business, accounting, programming, marketing, or whatever.

cowardlydragon• 10 years ago

Everybody who writes code does ETL in some form.

That's the fundamental action of computation. Read, compute, write.

This is a stupid article.

jessaustin• 10 years ago

Haha yes that's basically the Church-Turing Thesis right there.

AndrewUnmuted• 10 years ago

Most companies structure their data science departments into 3 groups:
Data scientists ... aka “the thinkers”
Data engineers ... aka "the doers"
Infrastructure engineers ... aka "the plumbers"

The author is clearly not an infrastructure engineer.

dllthomas• 10 years ago

Why? I have certainly described work I've done as "plumbing".

jxnl• 10 years ago

Which is why he spent 4 years doing managing data plat at Netflix...

systems• 10 years ago

i would bet that the author must be thinking that those who disagree with him are ... mediocre engineers or developers

personally, i think, there is nothing wrong with being average .. people with average skills built great things

mediocre is just a mean way to say average

nkurz• 10 years ago

ETL (Extract, Transform and Load) is a process in data warehousing responsible for pulling data out of the source systems and placing it into a data warehouse.

http://datawarehouse4u.info/ETL-process.html

spriggan3• 10 years ago

Thanks, I had no clue what ETL meant, but the article was probably not directed at my kind.

kelvin02• 10 years ago

What's an ETL? Electronic Transport Layer? Big Data N00b here ...

reverius42• 10 years ago

https://en.wikipedia.org/wiki/Extract,_transform,_load

wyldfire• 10 years ago

The original audience might not have been as broad as HN's, but it should've been expanded upon first usage IMO.

tyre2• 10 years ago

The article is about not over-engineering solutions to problems you do not have. If you don't have interesting problems that require world-class solutions, then don't hire as if you do.

If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”.

And then comes this line:

At Stitch Fix, we strive to be Best in the World at the algorithms and analytics we produce.

Without further justification, why does StitchFix, a subscription shopping service, need to be the "Best in the World" at algorithms and analytics? They have harder problems than Google or the Centers for Disease Control or NASA?

Unless they have justification for that, it seems a bit ironic given the article's ire for over-engineering.

noddingham• 10 years ago

I could have done without the first half of the post telling me that I (or others) are mediocre, then going on to tell me how the author (and his fellows) are not just because they strive to be the "Best in the World".

This just reads like a puff piece for another valley startup by some guy who's better than you. Oh, and here's how we do it, you should try doing it this way too, because we think it's totes the best.

trimbo• 10 years ago

I recommend it every time this topic comes up:

https://github.com/google/crush-tools

"Big" data on the command line.

jaz46• 10 years ago

Most of the comments in this thread are focusing on the author calling ETL boring -- that is the title after all. But I found the greater point of the article to be about empowering data scientists and giving them autonomy. This post reminds me of Jerry Chen's DDI post [1], except it's about data science.

The notion that a data scientist's only job is to "write a statistical model" and then it's someone else's problem to run it in a distributed environment only exacerbates the problem and lowers DS code quality.

Full disclosure: my company Pachyderm [2] is trying to solve exactly the problem Jeff is talking about in the post. We've built a data processing platform on top of the container ecosystem. Basically, the data scientist has complete control over the runtime environment for their analysis since everything is bundled into a container. It scales to work for actual "big" data, but it also great for small teams that don't have massive infrastructure resources.

[1] http://venturebeat.com/2015/04/01/the-geek-shall-inherit-the... [2] github.com/pachyderm/pachyderm

kpga• 10 years ago

Maybe a bit off topic, but from the 3 roles mentioned (Data Scientist, Data engineer and Infrastructure engineer) which one (if any) is better suited for working remotely?

georgewfraser• 10 years ago

I would take this a step further and say NOBODY should write their own ETL. In a world where:

1. SaaS services have APIs

2. Your database is hosted in the cloud

3. You use a standard SQL data warehouse that is also hosted in the cloud.

ETL from (1, 2) to (3) is a completely standard problem, and you should be able to buy a fully-automated solution. My company (Fivetran) does this as a service. We've replaced lots of homebrew data pipelines built by our customers, and we always see the same issues:

* Homebrew ETL pipelines use fancy big-data tech like Hadoop and Kafka in places where it has no relevance, like syncing your 20 GB Salesforce instance.

* Homebrew ETL pipelines don't deal with all the dark corners of the data sources, such as: what happens when someone adds a new custom column? What happens when your MySQL read replica fails over and a new binlog starts? Etc.

The lesson being, don't do this yourself.

jkestelyn• 10 years ago

All the angst about "big" data that "isn't big" is based on a false premise. "Big data" was NEVER just about scale, but rather intended to be equally descriptive of diversity as well as velocity. The problem is that whomever coined the term made the same strategic error as whomever coined "global warming" -- the adjectives used are too specific to adequately describe the full range of qualities involved.

dunkelheit• 10 years ago

In other words division of labor does not work quite so well for a data science department as for a pin factory. The proposed solution (letting data scientists code more) is not radical enough in my opinion. Why not muddle the roles even further? Let everybody feel the pains that people in the other roles experience. Foster empathy and personal connections. Let developers talk to the users and vice versa.

I worked at a company where distinction between the roles was emphasized by physical separation, presumably so that they won't interfere with each others day-to-day duties. The downside is that each group starts caring about their particular thing only, feeling that they are the ones who really keep the place running and other groups are bozos doing their job incredibly poorly.

chris_wot• 10 years ago

It's really amazing that there are businesses who collect a lot of data, spend most of their time transforming and loading it - then do nothing with it. Nothing effective at any rate.

I'm genuinely curious - to analyse data effectively, is there a baseline of statistical understanding you need to have? If so, what is it?

dgudkov• 10 years ago

I've done quite a few BI/DWH projects and what I found is that the best approach is to begin from deep prototyping done by data analysts and ETL developers together. It may all start with just a few spreadsheets and a simple dashboard. After many iterations and a lot of brainstorming it grows into a rather developed working prototype that the both sides have equally contributed to. Then the prototype is productionized by the engineers using standard ETL/whatever tools. So everybody gets the credit, and everybody is motivated. This experience made me create EasyMorph [1] -- a tool for quick ETL prototyping and brainstorming. It's like Excel but for tables, and it's equally suitable for data scientists and developers.

[1] http://easymorph.com

eveningcoffee• 10 years ago

This is the same pattern as architects, coders and administrators. So yeah, nobody wants to be a coder monkey and administrator role is also tedious, and the added value of architects is actually kind of low.

Sorry, I could not get any further than when the sale pitch kicked in, so sorry when there was anything new after that.

StudyAnimal• 10 years ago

ETL is just a small part of it, and engineers should probably have more of a role than they do. My last project was a data warehouse one, where the thing was obviously slapped together by PowerCenter users. They thought it was all about the ETL, They forgot the other 95%, how to engineer large scale, complex, maintainable software solutions.

Sure, let the powecenter users "write the ETL", but then they need to get the heck out of the way and let the big boys actually build the warehouse.

kakoni2• 10 years ago

How do people do ETL these days? Using spark? Some framework?

Personally for smaller projects I've used kiba[1] or transforms in pgloader [2]

[1] http://www.kiba-etl.org/ [2] https://github.com/dimitri/pgloader

mattexx• 10 years ago

The best-case outcome of many efforts of data scientists is an artifact meant for a machine consumer, not a human one.

I posit this outcome is absolutely necessary for any data science project to be worthwhile in any organization.

In the case where the project produces a report and goes no further, you still need the data and code for reproducibility, one of the main principles of the scientific method [1].

In the case where the project gets handed off to engineers to re-implement, reproducibility is even more critical, since the engineers best effort to reproduce the code will almost certainly not be successful the first time, and you will need to validate many versions of the production model. Doing this by hand even once it wasteful, doing so many times is tragically so.

In the case where the data scientists can produce a service worthy of production use, kudos!! But understand the caveat that in truly big data or big compute flows, this outcome remains highly unlikely.

[1] https://en.wikipedia.org/wiki/Reproducibility

scandox• 10 years ago

This reminds me of Spolsky's story about MS trying to create a Master Slave paradigm in coding. The Master would define functions and the slave would write the actual code in the functions. But of course no one wants to be the slave. Everyone needs to feel they are a thinker. Naturally.

Engineers Shouldn’t Write ETL

Related Stories