qoega

It is meant for single reader/writer workload so not meant to be used as a service

In ClickHouse it is just `INSERT INTO t FROM INFILE 'data.csv.gz'`. Any supported format, any encryption, autodetected from file name and sample data piece to get column types, delimeters etc. Separate tools to convert CSV are not necessary if you can just import to db and export as SQL Statements.

echo "name,age,city John,30,New York Jane,25,Los Angeles" > example.csv

clickhouse local -q "SELECT * FROM file('example.csv') FORMAT SQLInsert" INSERT INTO table (`name`, `age`, `city`) VALUES ('John', 30, 'New York'), ('Jane', 25, 'Los Angeles');

Are you guy comparing 16vCPU/32GB vs 8vCPU/32GB and say yours is only 1.6 faster?

Now you rarely use basic MapReduce primitives, you have another layer of abstraction that can run on infrastructure that was running MR jobs before. This infrastructure allows to efficiently allocate some compute resources for "long" running tasks in a large cluster with respect to memory/cpu/network and other constraints. So basically schedulers of MapReduce jobs and cluster management tools became that good, because MR methodology had trivial abstractions, but required efficient implementation to make it work seamlessly.

Abstraction layers on top of this infrastructure now can optimize pipeline as a whole by merging several steps into one when possible, add combiners(partial reduce before shuffle). It requires whole processing pipeline to be defined in more specific operations. Some of them propose to use SQL to formulate task, but it can be done using other primitives. And given this pipeline it is easy to implement optimizations making whole system much more user-friendly and efficient compared to MapReduce, when user has to think about all the optimizations and implement them inside single map/reduce/(combine) operations.

You can even migrate your zookeeper to ClickHouse keeper. It requires small downtime, but you will have all your zookeeper data inside and your clients will just work when your keeper will be back

Did not expect to see issue I created

Open-source ClickHouse also allows both real-time and large historical data.

I think atwong just promotes his product https://news.ycombinator.com/threads?id=atwong

It is nice to have an image of expected dashboard in readme.

Passion and experience

It would also be good to see GEOS, GDAL and PROJ integrated into ClickHouse.

You can just add feature requests like this to ClickHouse issue tracker. https://github.com/ClickHouse/ClickHouse/issues/45129

There is no need to use clickhouse-local for 'local' analysis. It just the same ClickHouse, but it will not store any local data. Just use it as ETL tool. Something like this will work: INSERT INTO FUNCTION s3('https://storage.googleapis.com/BUCKET/test_data/file.csv', 'XXXXX','XXXXX',CSV) SELECT col1, col2 FROM postgresql(postgres1, schema='schema1', table='table1') WHERE col1 > col2;

I don't know why exactly you need BigQuery in your scenario. If I wanted to have Postgres->ClickHouse it would take single INSERT SELECT from ClickHouse.

If you need just CSV result dumped to GCS you can use clickhouse-local mode that has all the features like integrations with Postgres, GCP, formats.

Only but it is not a service with UI where your data analysts can click and drag what they want to export. But SQL can be simple for them to write and you need nothing more than a trivial cron job analogue to run it.

Sure you can read from S3.

You can define table that will look at your files like this https://clickhouse.com/docs/en/integrations/s3/s3-table-engi...

Or just use s3 table functions https://clickhouse.com/docs/en/integrations/s3/s3-table-func...

You can use it as a command line tool running one query at a time or in interactive mode.

Can you provide SingleStore result for TPC-H and TPC-DS? I can't find it. Why? [1] https://www.tpc.org/tpcds/results/tpcds_results5.asp?orderby... [2] https://www.tpc.org/tpch/results/tpch_results5.asp?orderby=d... Disclaimer: I work for ClickHouse

I think optimizations that you focus on for self-hosted ClickHouse are the same as for Cloud. In self-hosted it helps to improve your throughput/capacity with fixed allocated resources. In cloud it directly affects cost.

For those complex pipelines you may find more useful to run tests during trial. Data distribution, partitioning and so on can change actual cost significantly so estimates can be too pessimistic or optimistic

License states the following. All other modifications are not standardized and you can't just compare systems. Otherwise there would be another standardized benchmark in the list you propose to run and publish.

c. Public Disclosure: You may not publicly disclose any performance results produced while using the Software except in the following circumstances: (1) as part of a TPC Benchmark Result. For purposes of this Agreement, a "TPC Benchmark Result" is a performance test submitted to the TPC, documented by a Full Disclosure Report and Executive Summary, claiming to meet the requirements of an official TPC Benchmark Standard. You agree that TPC Benchmark Results may only be published in accordance with the TPC Policies. viewable at http: //www.tpc.org (2) as part of an academic or research effort that does not imply or state a marketing position (3) any other use of the Software, provided that any performance results must be clearly identified as not being comparable to TPC Benchmark Results unless specifically authorized by TPC.

I think ClickHouse Cloud Beta addressed exactly this concern. It separates storage and compute and deals with scaling. There is no sharding so you don't need to deal with scaling.

You can't post results of TPC benchmarks without official audit. So it complicates posting results. You can't find common names that are usually compared with ClickHouse there [1]. So open standardized ClickBench tries to encourage benchmarking for everyone.

There are numerous benchmarks that use similar to TPC queries, but those are not standardized and can be misleading. For example a lot of work was done by Fivetran to get this report [0], but they show only overall geomean for those systems and you can't understand how they actually differ. Anyway their queries are not original TPC - variables are fixed in queries, they run first query when official query is a multiquery.

Contributors from Altinity run SSB with flattened and original schemas [2]. SSB is not well standardized and we see a lot of pairwise comparisons with controversial results - generally you can't just reproduce them and get all the results in single place for the same hardware.

[0] https://www.fivetran.com/blog/warehouse-benchmark [1] https://www.tpc.org/tpcds/results/tpcds_results5.asp?orderby... [2] https://altinity.com/blog/clickhouse-nails-cost-efficiency-c...

I may assure you that 1WU is not 1 part. Not even close. You can check it using trial credits with your data.

For example, I just checked that uploading 1.1GB example table(cell_towers with 14 columns) cost me 0.38 write units.

You can consider that ClickHouse allows both to query a lot of supported external data sources(s3/hdfs/mysql/postgre/...) and to store data in pretty efficient columnar way with compression, indexes and all the bells and whistles. Native storage allows to use all the information about keys/indices to build query plan faster. With trino you can't store data inside trino. You can't even insert data using trino which allows you to solve scenarios like 'readonly analytics'. Trino allows you to use single query language for all the supported systems. So if you have a zoo of DBMS and object storages that you can just query it can help you to hide this complexity.

Many queries that did do not run had aggregations over strings like MIN/MAX. I don't know specifics why many Java based DBMS lack this aggregation functions.

You can see that setup used is the one provided in a package: single-server/medium. It makes sense to improve setup, but I recommend to provide better configuration by default. I think it is a common courtesy that system should just work at most cases without hard tuning.

There are several existing benchmarks that test query optimisers with a lot of joins. It does not show performance of query engine, but more likely how good is your optimiser was tailored for this queries.

It is that big only with debug symbols. If you take packaged version of ClickHouse it will be much smaller. Or just strip large binary manually if you already have it.

Did you miss previous try of TimescaleDB to show that they are better than ClickHouse https://news.ycombinator.com/item?id=28945903?

It is partially true, but this benchmarks force schema. You can't reorganise data for example in wide table or add indices. So it actually does not show you how to use the system to solve this type of problems in a best way possible, but checks unoptimised results as if you never learn and never utilise best practices of the DBMS you choose for production.

Agree. And for a blog post it can even have a story like: "We compared with ClickHouse and we were 10x slower, than we looked at this case and made it 100x faster. Thank you, benchmark and ClickHouse developers that showed us use case where we could do better."

For me benchmarking is usual - "Why this query takes so long? We need to improve it. Sometimes 1000x times."

There is a small list of vendors that do not forbid to run benchmarks with their systems. https://cube.dev/blog/dewitt-clause-or-can-you-benchmark-a-d...

That is why there is a small subset of vendors that are being 'attacked' by this comparisons.

If you will use S3 as a storage for cold data it can be become just S3(or other object storage) cost. And ClickHouse compression rates for Twitter-like data can be 10x or better as you do not store raw data and data in stored in columns.

Concerning ingestion costs, I think it took me ~10 hours of 4vCPU vm last time, when I loaded 1.5 TB data to ClickHouse from this dataset: http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-ta... 1.1 billion rides became 3.5 billion already.