Show HN: An ETL engine with hard memory guarantees

https://github.com/logannye/emsqrt
by logannyeMD • 9 months ago
2 0 9 months ago

This is an external-memory ETL engine that enforces hard memory bounds (as opposed to systems that can only try to stay within limits). It's the result of playing around with a recent idea from complexity theory, tinkering to find practical applications of where efficient time-space simulation could be useful.

It enforces a strict memory cap via RAII tracking and scheduling, so you can process arbitrarily large datasets with a fixed, small memory footprint. Operators automatically spill to disk when needed. Might be helpful for serverless and edge environments where you need predictable resource usage.

Related Stories

Loading related stories...

Source preview

github.com