I'm working on Starlake.ai, an open-source data engineering platform that simplifies data ingestion, transformation, and orchestration. It's designed to make data engineers' lives easier while maintaining powerful control over data pipelines.
Key features:
- Declarative YAML-based configuration for data pipelines
- Intuitive web UI for those who prefer visual interfaces (see video section in https://starlake.ai)
- Native integration with both Airflow and Dagster
- No-code/low-code approach to data transformation
- Support for multiple data sources and formats
- Built-in data quality validation
- Automated schema inference and evolution
Data Processing Capabilities:
- LOAD: Ingest data from various sources (CSV, JSON, XML, Parquet, etc.)
- TRANSFORM: Write SQL transformations and test them in DUCKDB.
- Support for major data warehouses:Snowflake, BigQuery, Databricks, PostgreSQL, DuckDB for local development and small data
What sets it apart:
- Full Git Integration: All changes (whether made through UI or YAML) are automatically versioned in Git
- CI/CD Ready: YAML configurations can be directly integrated into your CI/CD pipelines
- Infrastructure as Code (IaC) friendly: Perfect for GitOps workflows
- Dual Interface: Everything possible in YAML can be done through the UI, and vice versa
- DuckDB Support: Perfect for local development and smaller datasets, allowing you to test your pipelines without cloud costs
The project is open source and we'd love to get feedback from the HN community. You can check it out at:
- Website: https://starlake.ai
- GitHub: https://github.com/starlake-ai/starlake
If you're dealing with data pipeline challenges or interested in modern data engineering tools, I'd love to hear your thoughts!