Show HN: New CI/CD Platform for Data Engineers

https://www.youtube.com/watch?v=-s-bULKecvY
by _ttnp • 3 years ago
4 0 3 years ago

In the last year I've become interested in how Data Engineering is evolving to become more like Software Engineering, with the adoption of source control, branching/merging, unit testing, modular code etc.

This has been made possible in part by dbt (https://getdbt.io), which is a really nice tool for defining and running data transformations.

I had the idea to build an Open Source CI/CD platform around dbt, to automate the process and create a defined path to production for data changes.

Though we could of course to do this with Jenkins or Github Actions, the thinking is that we can build a tool which avoids a lot of scripting when developing the pipeline, and provides a user experience that is optimised for Data Engineers, for instance exposing the underlying data as it is transformed and tested.

Here is an introductory blog post with more thinking - https://timeflow.systems/blog/introducing-timeflow

I would be interested in any thoughts on the idea and approach.

Related Stories

Loading related stories...

Source preview

youtube.com