Kore: Binary File Format Optimized for Modern Data Systems (Open Source)

https://github.com/arunkatherashala/Kore

by arunkore2026 • 2 months ago

12 9 2 months ago

arunkore2026• 2 months ago

A binary file format built from first principles for modern data systems. Parse 100MB 50x faster than JSON, with 50-70% better compression. Full language support (Python, Java, JavaScript, Go, C#, Ruby). Includes a VS Code extension for viewing .kore files. 3 years of production testing before open source release.

inheritedwisdom• 2 months ago

Curious what you see as key differentiators over parquet / iceberg formats with snappy or similar compression schemes?

theginger• 2 months ago

What does it do better than parquet? The compression ratio quoted is lower than parquet but I expect higher to be better in this context

microflash• 2 months ago

The fastest, most compressed columnar format for big data

How large a dataset can it tackle? I work with Parquet files spanning 300million+ records (~800MB files) using DuckDB and it works within seconds.

I might be interested to see benchmarks against Parquet and Vortex. A DuckDB extension would be great as well.

arpadav• 2 months ago

looks like a cool project, but id say keep working on it since there seems to be some confusion on why someone would want to use this: no benchmarks and overall pretty vibe-codey (which id personally be very hesitant to use in production)

another comment already mentioned comparison to vortex, which is the same compression ratio and same speeds as youre claiming - but your compression is half of parquet. and if speed is the main goal youre going for, python is an interesting choice. no hate, but def keep working on it, and would love to see more concrete benchmarks with various columnar store types

miohtama• 2 months ago

How it achieves the performance? What is the tradeoff?

saulpw• 2 months ago

47 releases in 3 weeks. Dozens of meaningless cruft files committed at the project root, like "h .kore_fileformat_source_v0.1.0.zip -Algorithm SHA256" with a summary of "less" commands. 19 deployments, all failed.

A+++ pure slop. If there's any value in this repo, I couldn't find it after a cursory examination, and I'm not willing to expend the effort to continue looking.

pkhuong• 2 months ago

kore-private-key.asc https://github.com/arunkatherashala/Kore/blob/release/v0.1.0... is pretty good too.

yogorenapan• 2 months ago

I thought HN was restricting Show HN posts for new accounts now. I suppose not? I avoid clicking into these posts nowadays because so many are just slop

Kore: Binary File Format Optimized for Modern Data Systems (Open Source)

Related Stories