Summarizing Data in SQL

http://blog.yhat.com/posts/summarizing-data-in-SQL.html

by elisebreda • 10 years ago

36 9 10 years ago

blog.yhat.com

d33• 10 years ago

Nice! Does anybody know a project that combines this kind of techniques in a program that could be ran across billions of rows, most preferably something that scales to multiple nodes?

TheLogothete2• 10 years ago

MPPs do this. Azure Data Warehouse, Amazon Redshift, Google BigQuery.

jmsmistral2• 10 years ago

Doesn't have to be a DB-as-a-service though - Oracle and Greenplum are MPP, for example.

vgt• 10 years ago

Shameless plug:

https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-...

evook• 10 years ago

Every database I could think of at the moment. Even SQL-Lite is able to handle billions of rows.

Probably some xargs sed awk hacks are also sufficient for results in milliseconds.

You start getting problems when dealing with >128 TB databases like the legacy crm databases at Bayer and Siemens e.g.

jmsmistral2• 10 years ago

I think ">128TB" is stretching the upper-limit for single-node database. When you're dealing with > 10TB in a single node, it's probably wise to 'shard' or partition the data over more than one node (think backups/archiving, query performance,...).

To answer the initial question - take a look at CitusDB that runs over PostgreSQL. They seem to be doing some cool work in real-time analytics/scalability: https://www.citusdata.com/

frik• 10 years ago

Percentile
In MySQL, we can use local variables to keep track of the order (...) Here's the Postgres version.

I would be interested in the MySQL percentile snip.

alfanick• 10 years ago

Cool article, but please update code snippets to some consistent codestyle :)

Summarizing Data in SQL

Related Stories