carlineng

This is really a question of economics. The biggest organizations with the most ability to hire engineers have need for technologies that can solve their existing problems in incremental ways, and thus we end up with horrible technologies like Hadoop and Iceberg. They end up hiring talented engineers to work on niche problems, and a lot of the technical discourse ends up revolving around technologies that don't apply to the majority of organizations, but still cause FOMO amongst them. I, for one, am extremely happy to see technologies like DuckDB come along to serve the long tail.

Not yet, but I believe the DataCouncil staff recorded it and will post it to their YouTube channel sometime in the next few weeks: https://www.youtube.com/@DataCouncil/videos

I just watched the author of this feature and blog post give a talk at the DataCouncil conference in Oakland, and it is obvious what a huge amount of craft, ingenuity, and care went into building it. Congratulations to Hamilton and the MotherDuck team for an awesome launch!

I agree 100% that this needs to be more of a thing. For data engineers building data pipelines, queries are like functions, and table schemas are like types. There needs to be a way to write a query that runs on an abstract interface, rather than an actual table. To do this, most folks rely on string templating in Python or Jinja, which makes the development process really cumbersome. As a result, most teams end up in scenarios where data pipelines are always a big mess of spaghetti SQL, or they are stuck maintaining complex frameworks that abstract away common logic, but are inscrutable to the average user.

I wrote a longer blog post about this recently: https://carlineng.com/?postid=holy-grail-data-engineering#bl...

The argument is more than that -- namely that in addition to having sophisticated AI, Google also controls the OS (Android) and hardware (Pixel). Being able to integrate best-in-class AI at every level of the stack is a tremendous advantage. OpenAI can't do this because they don't control the OS, and will always need to go through an app. Apple can play since they control OS and hardware, but at the moment they appear pretty far behind in the AI aspect.

My favorite discussion of this topic is from David Beazley: https://www.youtube.com/watch?v=5C6sv7-eTKg

He does a wonderful job of taking very dense mathematical notation and explaining it in ways that anyone can understand. He derives the basic concepts of the lambda calculus from the ground up using Python. Super fun to follow along with.

My understanding of Cube is that iterating on the data model requires the user to (1) write SQL to develop a metric (2) edit YAML or JS config to incorporate the new metric (3) issue API request to Cube server and (4) compare results to raw SQL. Am I mistaken? Does Cube offer a smoother way to do this exploration/iteration?

Thanks for the kind words! Admittedly, a lot of the stuff you're asking about is still a work in progress, and we don't have good answers for it all just yet. The upside of that is you have the potential to influence which direction we take next. If you're thinking about building on top of Malloy, join our community Slack channel, and we'd be happy to provide guidance or take your suggestions/feedback! https://join.slack.com/t/malloy-community/shared_invite/zt-1...

Glad you asked! It is certainly usable for writing apps. We publish an npm package, and the Malloy VSCode extension [0] is one such example of an app built on top of it. There's also a demo of a toy CLI app that showcases the simplest possible use of the SDK: https://github.com/malloydata/malloy-demo-bq-cli

Fair critique on the presentation. That's mostly a function of the circles I run in, which is heavily weighted towards data analysts, data engineers and "analytics engineers". But to your point, I think that cohort is unlikely to be early adopters of a tool like Malloy, and that developers are a much easier sell.

[0] https://github.com/malloydata/malloy-vscode-extension

Semantic layers, also known as metrics layers or “headless BI”, have become a popular topic in the online data community in recent years. For all the hype, the idea hasn't seen much traction. In this blog post, I hypothesize why not, and describe the Malloy language and why I think it has a better chance at succeeding.

I've written quite a bit about a new query language called Malloy: https://carlineng.com/?postid=malloy-intro#blog

It's a semantic layer with an integrated query language that makes a ton of improvements on SQL. Full disclosure, I actually joined the team a couple of weeks ago :)

Would love to! How do I get in touch? My contact info is in my profile.

I'd wager you're right. All the "boring" stuff that's actually very complicated/difficult, and without which no large enterprise will adopt a technology.

MotherDuck has been making the rounds with a big funding announcement [1], and a lot of posts like this one. As a life-long data industry person, I agree with nearly all of what Jordan and Ryan are saying. It all tracks with my personal experience on both the customer and vendor side of "Big Data".

That being said, what's the product? The website says "Commercializing DuckDB", but that doesn't give much of an idea of what they're offering. DuckDB is already super easy to use out of the box, so what's their value-add? It's still a super young company, so I'm sure all that is being figured out as we speak, but if any MotherDuckers are on here, I'd love to hear more about the actual thing that you're building.

[1] https://techcrunch.com/2022/11/15/motherduck-secures-investm...

I had a student storm out of my class [1] because I was teaching relational databases instead of web3.

Wow... that exchange is surreal. I wonder if it was some kind of fraternity prank or troll job?

[1] https://www.youtube.com/watch?t=1490&v=uikbtpVZS2s

My hope with the post is to convince you that both the syntax and semantics of Malloy are more than just different, and indeed are actually better.

In the article, the SQL example results in output that cannot be used safely by downstream queries (i.e., it breaks algebraic substitution). This is generally true of any query in SQL that tries to aggregate along multiple levels of granularity. Malloy solves this limitation by making "nesting" of relations a first-class part of the language.

This blog post looks at SQL from the lens of data analysis, and provides a concrete example where it is seriously problematic.

The post itself provides a very real example of a flaw with the SQL language beyond simple aesthetics. Namely, that typical SQL queries only represent data in rows and columns, yet data and queries are often multidimensional. Any query that must represent data aggregated at different granularities produces output that is difficult and dangerous to use.

I think an analogous technology to Malloy is Typescript. It provides very real benefits over Javascript, but I don't think anyone would argue that it's 10x better. There will always be a lot of Javascript (and SQL) in the world.

This is a follow up to my last post comparing SQL to the new Malloy query language, and a response to some of the HN commentary I received around it.

My last submission [0] generated some interesting discussion that motivated me to think a lot more about exactly why SQL's syntax is problematic. This post explores that idea in more depth.

[0] https://news.ycombinator.com/item?id=32738874

To help understand the results of the benchmark, I find it helpful to look at how the benchmark is constructed, and what it tests for. From the README:

"The dataset is represented by one flat table. This is not representative of classical data warehouses, which use a normalized star or snowflake data model. The systems for classical data warehouses may get an unfair disadvantage on this benchmark."

Taking a look at the queries [0], it looks like it mostly consists of full table scans with filters, aggregations, and sorts. Since it's a single table, there are no joins.

[0] https://github.com/ClickHouse/ClickBench/blob/main/snowflake...

I remember Tim Cook said they are not in the Ads Business. Unfortunately I am too busy, Apple's PR has worked hard to delete those tracks of what Tim Cook said, or Google's search engine is longer showing what I am looking for.

The first sentence of the actual article links to when Tim Cook said that.

<img> tags use the loading="lazy" property which doesn't actually load the image until it's displayed. Everything else is just text, which is super tiny.

https://github.com/cadars/john-doe/blob/4f5a3eafecbdffa9d34f...

I really like the John Doe template. Everything is in one HTML file and one CSS file: https://github.com/cadars/john-doe/

From the readme: “This setup is databaseless, javascriptless, and buildshit-free, so you can edit your website with a text editor and upload it somewhere like a normal person.”

I would categorize it differently. Rather than just objecting to their policy, he is describing a reason why current policies are suboptimal, and suggesting ways that current policies could be improved. Namely, that domestic central banks around the world don't coordinate policy with foreign central banks, despite the existence of major feedback loops between foreign and domestic policy. He hypothesizes that the world economy would be better off if there were more global coordination on monetary policy.

Just to name a few off the top of my head --

- Data type mismatches between systems

- Differences in handling ambiguous or bad data (e.g., null characters)

- Handling backfills

- Handling table schema changes

- Writing merge queries to handle deletes/updates in a cost-effective way

- Scrubbing the binlog of PII or other information that shouldn't make its way into the data warehouse

- Determining which tables to replicate, and which to leave behind

- Ability to replay the log from a point-in-time in case of an outage or other incident

And I'm sure there are a lot more I'm not thinking of. None of these are terribly difficult in isolation, but there's a long tail of issues like these that need to be solved.

I wonder if the underlying technology is from their acquisition of Alooma [1] several years back. High volume change data capture from an OLTP database to a data warehouse is a problem that seems simple on its face, but has a lot of nasty edge cases. Alooma did a decent job of handling the problem at moderate scale, so I'm cautiously optimistic about this announcement.

[1] https://techcrunch.com/2019/02/19/google-acquires-cloud-migr...

I've written about this topic a fair amount; first examining some of the criticisms of SQL that have been present since the language's inception [1], then looking at a new project called Malloy that I'm quite excited about, and think has a lot of potential to address some of the problems of SQL as it relates to data analysis [2].

[1] https://carlineng.com/?postid=sql-critique#blog [2]: https://carlineng.com/?postid=malloy-intro#blog

Aesthetics are largely subjective. Even though I like Malloy syntax better than SQL syntax, the majority of the actual post is dedicated to reasons above and beyond syntax that I think Malloy has promise; specifically, its integration of a semantic layer into the core of the language.

Nope, those are the primary components of a semantic layer. Most other semantic layer products have three main components: input data sources -- typically SQL queries or table names, configuration -- the semantic layer describing all the things you mention above, relative to the input data sources, and the access layer -- usually a non-SQL API that consumers must use to consume data that has been modeled by the semantic layer. Check out the docs for Cube [1] for an example of this. Cube also has a SQL API, but it's not fully fleshed out yet.

What makes Malloy shine is that all 3 of these things are integrated in the same language, so users don't have to jump between different tools to model and explore their data. You can query/explore data in Malloy, iterate on functions to express your business logic, and immediately view the results. Doing this in something like Cube would require you to: (1) write SQL queries to prototype the function, (2) update the Cube configuration files with your changes, and (3) hit the REST API with a request to view results. In Malloy, it's all just writing and running Malloy queries.

[1] https://cube.dev/docs/query-format