ryguyrg

(Transparent: I’m one of the co-founders of MotherDuck)

Would love to hear your feedback!

note: i am the author of this blog post, and a co-founder at MotherDuck.

note that DuckDB allows that reverse ordering (FROM-first)

FROM table SELECT foo, bar WHERE zoo=“goo”

wohoo! glad you noticed that. Hamilton is amazing.

ROFL

In DuckDB UI and MotherDuck.

Awesome video of feature: https://youtu.be/aFDUlyeMBc8

Disclaimer: I’m a co-founder at MotherDuck.

We updated the video [if that's the reference], because it is not yet open source. Thanks for pointing that out!

i’m one of the co-founders at MotherDuck. our team is building the UI in collaboration with the team at DuckDB Labs.

this is a first release. we know there are going to be tons of feature requests (including @antman’s request for simple charts). feel free to chime in on this thread and we’ll keep an eye on it!

meanwhile, hope you enjoy this release! we had lots of fun building it.

you can scale vertically with a much better tech than awk.

enter duckdb with columnar vectorized execution and full SQL support. :-)

disclaimer: i work with the author at motherduck and we make a data warehouse powered by duckdb

Co-founder of MotherDuck here. Let us know if you have any feedback or questions -- we're super excited to release this as a valentine's day present to all those data explorers out there :-) Hopefully you agree it's as eggscellent as we do!

wait @tino, maybe we should call our support team the Pro Ducks team.

spot on!

we actually have a panel discussion on this at coalesce featuring matt housley and others. https://coalesce.getdbt.com/agenda/panel-discussion-fixing-t...

dbt folks actually brought this idea to us because it just makes so much sense -- no translating of sql between dev and prod.

highlights from panel brainstorming from winnie @ dbt: Local development: should be fast, representative but not comprehensive of prod data, and independent of cloud services if I want that.

The alternative has been that people typically use a schema in their production platform for development, CI workflows, etc. Costly and cloud-dependent.

DuckDB and its role in facilitating the MDS-in-a-box pattern has been a game changer for local data development, which ramped up the ability to share data, modeling work, and analysis the way we share open source tooling, but syntax remained an issue for deployment.

ryan (co-founder) here.

calling ourselves motherduck means we can either embrace the duck, or shy away from it. we've embraced it.

tino's board slides do refer to it as the Produck team :-)

Thanks for sharing - we're super excited about this launch! :-) [motherduck co-founder]

(i work at motherduck)

over my career i’ve seen countless cases where the complexity of a distributed system introduces as many (or more) reliability concerns as it solves. when everything works well, it can be beautiful. but when things go wrong, they can quickly become a cascading mess of hard-to-debug issues.

Just to agree with @orthozerox, it's not a 'killer feature' but about lowering the friction.

Lots of users: (a) mentally align better with SQL than pandas APIs, regardless of whether they know both or not (b) want decent performance on their analyses, which they aren't getting from pandas, and won't get from many OLTP-databases they're using over-the-wire (c) want ease of accessing parquet and csv locally and remotely with minimal development overhead. it's super simple with duckdb.

Nonetheless, some of the other things you pointed out are some tradeoffs. We're building a serverless cloud capability at motherduck on top of duckdb in order to address some of these and optimize compute and storage based on data locality, bandwidth and the need for collaboration amongst multiple people.

I'd love to understand the shape of this data and some of the types of queries you're performing. It would be very helpful as we build our product here at motherduck.

I have no doubt that there are situations where the cloud will be faster, especially when provisioned for max usage [which many companies do not]. However, there are a lot of these situations even where the local machine can supplement the cloud resources [think re decisions a query planner can make].

Feel free to reach out at ryan at motherduck if you want to chat more.

Okay, Google as a company and as a product is definitely in the top 1%, or top 0.0001% where big data drives profitable and bending changes ;-)

Agreed. But how many executives will agree to take these fine-grained actions to achieve value from the data? How many data teams are able to build up a strong-enough argument to convince them?

I've worked on many product-led-growth initiatives in the software industry. The software industry is probably the biggest 'believer in data' there is -- many scientific-forward minds who understand the value. However, even in the software industry, it's really hard to convince folks that if you make 5 improvements that net 1% conversion gain each, you can dramatically improve revenue.

Correct, much data has no value. The cost for storing the data, maintaining the data [in the day-and-age of privacy requirements especially], and combing through the data is often much greater than the value obtained from the data itself.

The expertise we need in the industry is people who understand applications in-and-out and make great decisions on what data is worth keeping for present and future applications. And what data is needed to be kept, but only in aggregates (or anonymized, which reduces costs of maintenance)

DuckDB is the open source project, based on the MIT open source license.

DuckDB Foundation is a membership-based organization to push the awareness, adoption and development of the DuckDB open source project.

DuckDB Labs is a commercial entity spun out of CWI that provides consulting services around DuckDB. The founders of this organization are the original creators of DuckDB [Mark and Hannes].

MotherDuck is a VC-funded startup that has a formal partnership with DuckDB Labs to develop and market a serverless analytics platform using DuckDB.

Hope this helps!

Lending Club has also done a lot of work in this area: https://github.com/LendingClub/mercator https://www.youtube.com/watch?v=xMsx5pS_VNo

And Sacha of Lyft was previously using similar techniques on the red team at Microsoft.

The awesome Bloodhound project productizes some of the red team use cases: https://github.com/BloodHoundAD/Bloodhound/wiki

Definitely check out this import script by @nicolewhite to import this data into Neo4j (until ICIJ releases it soon!)

https://gist.github.com/nicolewhite/f1980f01dc4342b369a6b3b6...

(disclosure: i work for neo4j)

This paper is a bit unrelated to Neo4j. Neo4j is an ACID-compliant native graph database. It is not a "specialized graph analytic engine."

Neo4j stores the graph data on disk (and caches in memory) as nodes and relationships. After index lookups to find the start points in the graph, all traversals of relationships are done in constant time -- allowing it to scale with linear performance characteristics, regardless of the size of the graph.

The referenced paper cites two main reasons that RDBMS would be better for the analytics use cases: (1) ability to express graph queries in SQL and (2) performance of executing those queries.

(1) Cypher is, like SQL, a declarative language. However, it represents graph constructs in a much more natural way -- "ASCII art for graphs." There's significant praise from developers on the web of the benefits of Cypher for traversing graphs, which is why we decided to open up the language: http://www.opencypher.org/

(2) As Neo4j isn't really intended as an analytics engine, its performance characteristics are not included in this paper. However, (expensive) indexes do not need to be created and maintained for every relationship in Neo4j. Similarly, these indexes do not need to be accessed for traversal (also expensive).

Sorry, we hit EC2 instance limits. Now fixed. Try again :-)

I agree if you're modeling a symmetric friendship network like Facebook. However, Twitter following can be asymmetric. I could store the directionality on a relationship property, but that gets even uglier.

FYI - you can technically upload directly to BigQuery and not go through cloud storage. It just tends to be a bit more error-prone in normal implementations. We're working on documenting resumable upload to handle error cases better.

The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_...

@iskander - do you have other approaches for uploading which you'd prefer?