š LDP Tax Pipeline ā Spark Declarative Pipelines on macOS (Without Databricks)
Excited to share my latest hands-on implementation of a LakeFlow Declarative Pipeline (LDP) built locally using Apache Spark 4.1 Declarative Pipelines ā running entirely on macOS, orchestrated with a simple cron job, and following a production-style Medallion Architecture.
š¹ What this pipeline demonstrates:
š„ Raw tax data ingestion from AWS S3
š„ Bronze layer for raw structured ingestion
š„ Silver layer for cleansing & enrichment
š„ Gold layer for aggregations & analytics
š¾ Persistence to Azure Data Lake Storage Gen2
ā±ļø Fully scheduled using cron (no external orchestrator)
š« No Databricks runtime ā pure Spark 4.1 SDP
Check this medium article. https://medium.com/@bijumathewt/lakeflow-declarative-tax-pipeline-using-apache-spark-4-1-64c965914c3...
