๐ LDP Tax Pipeline โ Spark Declarative Pipelines on macOS (Without Databricks)
Excited to share my latest hands-on implementation of a LakeFlow Declarative Pipeline (LDP) built locally using Apache Spark 4.1 Declarative Pipelines โ running entirely on macOS, orchestrated with a simple cron job, and following a production-style Medallion Architecture.
๐น What this pipeline demonstrates:
๐ฅ Raw tax data ingestion from AWS S3
๐ฅ Bronze layer for raw structured ingestion
๐ฅ Silver layer for cleansing & enrichment
๐ฅ Gold layer for aggregations & analytics
๐พ Persistence to Azure Data Lake Storage Gen2
โฑ๏ธ Fully scheduled using cron (no external orchestrator)
๐ซ No Databricks runtime โ pure Spark 4.1 SDP
Check this medium article. https://medium.com/@bijumathewt/lakeflow-declarative-tax-pipeline-using-apache-spark-4-1-64c965914c3...
