Dismantling ELT: The Case for Graphs, Not Silos

https://jack-vanlightly.com/blog/2024/11/26/dismantling-elt-the-case-for-graphs-not-silos

by sebg • 2 years ago

35 15 2 years ago

jack-vanlightly.com

smitty1e• 2 years ago

It's a great theoretical point.

The tools, the practices, all built around the premise that software and data teams don’t work closely together.

In particular, the various teams, even the same team with itself, end up being separated along a timeline.

Information Technology ends up being the embarrassment we all face that our data are rarely, if ever, showing up dressed for work.

jacknews• 2 years ago

Since the article uses lots of big words, phrases and memes (when a few simple words would suffice), and assumes you know it already, Conway's law is simply that your technical architecture will reflect your social/organizational architecture.

orbat• 2 years ago

"Big words, phrases and memes"? You got this distressed about an article using industry standard terminology for its target audience and assuming that you'd read (or would read) the post it linked to on Conway's law?

arkh• 2 years ago

provide “data APIs” to data teams

I feel like I'm reading part of Data Mesh again.

robertlagrant• 2 years ago

Hah. That is a good reference. Never seen a concept boosted so much.

dallasg3• 2 years ago

I hate to say it, but good documentation is the key here. Visualizing data as interconnected nodes breaks down silos, aligns teams and makes it easier to build reusable, loosely-coupled systems.

tspann• 2 years ago

reminds me nifi plus kafka

datatrashfire• 2 years ago

I missed the part where there was an actionable takeaway about how I as a practitioner am supposed to differently.

baq• 2 years ago

Conway’s law works both ways: if you want a system to be designed in a certain way, you create an organization who is building it in that way.

So, if you want a data mesh, build a mesh org. Not the easiest job.

chevman• 2 years ago

Been in BigCo land for 20 years now, and have seen the rise and fall of quite a few data/analytics/BI/reporting/AI/ML fads.

Honestly the whole landscape seems broken and unproductive at this point.

Countless vendors, platforms, cloud environments, industry/technical jargon - all with different pricing models, SLAs, tooling, etc etc.

Getting anything usable is a challenge and most orgs spin in a never ending cycle of data integration/normalization work that produces little business value.

My advice to teams now is simplify, reduce, streamline - get to the kernel of what you think you need and protect it all costs. Most of the shiny new objects being pitched as silver bullets are just ways for other people to make money off your margin.

heisenbit• 2 years ago

The article does not contain the words performance and security.

ericyd• 2 years ago

This comment does not contain the word therefore.

up2isomorphism• 2 years ago

Yeah, 100 people can literally implement what he is saying in 100 different way.

AIorNot• 2 years ago

As a 20 year veteran the biggest opportunity in new data space is potential for LLM unstructured data parsing and intelligent api integrations to break silos

This article assumes the business has a logical progression for its plans in question and then makes logical and somewhat rational decisions that lead to a operational estate vs a data estate when in realty it’s about budgets and pet projects and c suite whims and frank got fired after he brought in the last data engineering team who ended up building a data lake with a dashboard that one VP uses and know one else understands

Something gets built =>based on a limited dev budget it leads to something else getting built on top of it maybe if a consultant sells it well to an exec - ultimately it all becomes a complex spaghetti house of cards… or “data mesh”

naijaboiler• 2 years ago

I flat out disagree with the premise of that article. I think it is written from the perspective of an engineer that is primarily in the operational domain and only see the value of analytics domain as a supplier of data to the operational side.

The the truth is there are many many uses of data that isn’t transactional/operational and frankly have no need to be real time at all. So all the building and maintaining of graph network to deliver near real time data is just an unnecessarily over-engineered and unnecessary and costly complexity.

If your operational use cases are indeed near real time, please find some optimal way to do that. But don’t you dare assume many other data use cases need that

Dismantling ELT: The Case for Graphs, Not Silos

Related Stories