Show HN: Declarative Spark Transformations with Hamilton
https://github.com/DAGWorks-Inc/hamilton/tree/main/examples/spark/pysparkI'm really excited about this new feature in Hamilton and wanted to share. It allows you to express a series of spark transformations as a DAG of individual functions that can be managed at a column level, while running them efficiently on spark dataframes.