Guide to data tools landscape for developers

https://sinja.io/blog/data-landscape-guide-for-developers

by OlegWock • 6 days ago

134 36 6 days ago

Firfi• 6 days ago

Now I'll be thinking of "L" in ETL as "Land" and not "Load". Although the article doesn't propose that but uses a lot of "Land" terminology. "Load" => "load where? or FROM where?" - ambiguous "Land" => "land where?" - clear

nadzzz• 6 days ago

always understood "land" as the raw data layer though, i.e only the bronze layer in the article.

madsaylor• 6 days ago

Data is the new oil

hermitcrab2• 6 days ago

Both require a lot of refining, before they are any use. ;0)

jpitz• 6 days ago

Apache Avro has 2 encodings: binary AND json.

macintux• 6 days ago

Are you sure? I can't find any reference indicating that.

Update: Huh, TIL https://avro.apache.org/docs/%2B%2Bversion%2B%2B/specificati...

jbonatakis• 6 days ago

A data warehouse on the other hand is an OLAP database and is optimized to work on columns

A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.

datadrivenangel2• 6 days ago

You are pedantically correct but technically wrong, as even optimized postgres is going to suffer on analytical patterns without extensions. With extensions (DuckDB or Citus) you can do large aggregations, but regular postgres at medium/large scale (billions of rows, 100s of GB) starts having a lot of foot guns and complex babying to do analytics. A bunch of indexes and you'll be fine though.

antonvs• 6 days ago

But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL.

Other than money and time.

MNeverOff• 6 days ago

It's a good all-round primer, well written. Would love to hear more about larger-than-memory tasks and running local Dask clusters. I processed many-a-dataset that way that would normally make pandas choke.

michaepf• 6 days ago

This was great, thanks for writing it up. Even as someone in the data space for a long time, I learned quite a bit.

chrisweekly• 6 days ago

Great post! Also, I dig your site -- it's attractive and highly usable, and the "personal" toggle in the footer is a clever affordance I haven't seen before for separating professional content.

svfat• 6 days ago

Good to finally know the difference between a lake and a warehouse

botswana99• 6 days ago

Well, I find this post looks good, but a like lot of 'data for developers' posts it's just a list of tools. As if a collection of tools banded together actually makes your customer successful. What's missing? 1. There's nothing about deployment. How do I take this collection of tools and code and actually deploy it into production, or actually regression test it functionally? How do I make a small change in a database table and not have a massive regression? How do you do that automatically? How do you do it quickly? 2. It's cursory on testing. One of the biggest differences from a software developer to a data engineer is that your data providers give you crap data all the time. It could break. How do you test data? How do you get adequate test coverage? These things are essential for software developers and are actually doubly essential for data engineers and building analytics systems. 3. It's what success looks like. It's not just about a collection of tech; it's about making your customers successful. What does it mean to deliver good insight? How do you do it? How do you measure customer success, and measure your success? As a team, you wouldn't talk about software engineering without mentioning DevOps or DORA metrics. There's nothing here about that.

sigh

kingjimmy• 6 days ago

excluding Denodo from the list sows this is more of a non-enterprise guide to data management and tools. There is only one real semantic layer that can cover operational and historical data and thats Denodo. If you use snowflake horizon or unity, all the data needs to be loaded first and not real time.

aleda1452• 6 days ago

I have never heard of Denodo before. "The only real semantic layer" is a bold claim. From the website it just looks like another data governance tool?

flarco• 6 days ago

Nice guide. One addition in the ingestion section: sling (https://slingdata.io/). it's a single go binary, cli-first.

doobiedowner• 6 days ago

What about PI?

aleda1452• 6 days ago

As a data engineering person I can say that this is a great write up!

Some thoughts:

A "bubbling" topic right now is conversational analytics (i.e. talk to your data). There has been an explosion of tools in the last 6 months. YC is backing one too: https://getnao.io/

I feel like pandas is also somewhat frowned upon, the industry has moved on from that. Most SQL tools can now do everything that we could only do with pandas.

In my network everyone is talking about DuckDB. As long as you are under a 1TB it will have everything you need. I think most people should start with that vs locking themselves into something like Snowflake

sdpy2• 6 days ago

Just to add, people in my network have been talking about polars (as an alternative to pandas) and other dataframe libraries. They're much easier to use now thanks to the Narwhals compatibility layer (for example, Narwhals was recently added as a dependency to scikit-learn).

gwerbin• 6 days ago

What's different between Narwhals and Ibis? Why does the former exist when the latter has already (struggled along) existing for a while? Narrower scope / benefit of hindsight? No support for "cloud" data frames like Ray and Spark?

Also Pandas is very much still a great tool and it's only getting better. It has some fundamental limitations that are relevant for processing bigger datasets or running things with higher performance. But it's still my preferred data frame for interactive day-to-day work. I only switch to Polars (or DuckDB) when I want to maximize performance.

estetlinus2• 6 days ago

I love what I am hearing. I still see a lot of engineers using pandas, but it is such a horrible tools. You usually find an abandoned notebook with 100s of ”df_final_2” with sequential wrangling, making it impossible to understand what’s happening. Notebooks are also horrible for the same reason IMO.

With chat-your-data you have Hex, Claude + MCP, snowflake, Databricks etc… everyone’s in on it.

datadrivenangel2• 6 days ago

"Popular metadata catalog solutions are Hive Metastore, AWS Glue Data Catalog, and Unity Catalog from Databricks.

For query engine you can use, for example, Apache Spark, Trino, or Amazon Athena."

DuckDB is eating the query engines and catalogs. Really could use more coverage on how DuckDB is changing the data tools landscape.

estetlinus2• 6 days ago

Can you suggest any good resources for your statement? I know DuckDB is hyped, but didn’t know they’re eating engines and catalogs.

porksoda• 5 days ago

It is a POS. Try connect a read-only while another proc is connected, you cannot. Try have separate processes cooperate, db corrupts if open too close on heels of close. I hate ddb with the power of a thousand sharp wet farts. I bitched to my new boss and no sympathy, imagine the schadenfreude when a week later I heard the whole team move to postgres - the boss hit his own head on the thing.

ozhero• 6 days ago

This is an excellent well written article with just the information you need to get a basic understanding of the field.

stronglikedan• 6 days ago

beep boop

jrhizor• 6 days ago

Really a great layout of the space! I like how it doesn’t try to enumerate all options.

otter-in-a-suit• 6 days ago

This is a fantastic primer.

A few suggestions:

- Add a "last updated" note, since this space changes often (see the prefect/dagster situation that just happened)

- Add a note about MCPs and other LLM-driven tools and features are becoming more and more important (e.g. hex.ai or the various MCPs shipped with some of the tools you mention, such as OpenMetadata)

- Maybe organize the various tools you mention briefly by their license/model (lots of them can be self hosted, some are SaaS only), since a fully self-hosted data platform is (at least for now...) very much feasible

I also wish more people would talk more about the "engineering" part of "data engineering". I've seen way too many people who claim a title like "data engineer" but lack the fundamentals of building software and are really just copy-paste scripts together.

What I'd love more DEs to think about are things like {unit,integration,e2e,performance} tests, deployments, infrastructure, networking, monitoring (you do touch on that), and all the other things a regular SWE is expected to have at least basic competency in at a certain level. For instance, tools like dbt natively support tests, but people need to write them. Or how you don't have to click-ops Airbyte, there's a terraform provider etc.

vishnuharidas• 6 days ago

+ Also a few diagrams showing the end-to-end data flow and where all these tools fit in.

hermitcrab2• 6 days ago

Interesting summary. I'm surprised you didn't mention any desktop ETL tools (such as Alteryx, Knime, Easy Data Transform).

traceroute66• 6 days ago

Interesting write up, but what it really needs is a nice task vs tools table at the end as a "conclusion".

Someone reading the blog the first time might well be willing to read through the large amount of text.

But returning to it at a later date ? A quick reference TL;DR table would not go amiss IMHO.

penguin_booze• 6 days ago

Who is this article for

WHOM

sp_from_db• 6 days ago

Fantastic primer. This should be required reading for all new devs.

Curious about your thoughts on some of the new architectures like vector lakebase or LTAP. What do those mean for developers?

di1eep• 5 days ago

This is a great article covering the ecosystem but has not covered one of the key advancements in this area in the past 40 to 50 years., which is IVM (incremental view maintenance).

The topic could start from database “views” and how analytical queries are basically different views of the same underlying data and how IVM solves the batch+stream unification and real-time queries over data elegantly. Tying change data capture into the picture as well.

The company “Feldera” and its founding folks are the leading researchers/experts on IVM and it’s becoming increasingly relevant for all kinds of workloads.

Guide to data tools landscape for developers

Related Stories