Hey everyone,
I recently worked on building a modern financial data lakehouse using Spark Declarative Pipeline OSS (SDP OSS), Apache Iceberg, and AWS Glue Catalog.
The blog covers:
- Building declarative data pipelines with Spark
- Using Apache Iceberg as the table format
- Managing metadata with AWS Glue Catalog
- Streaming + batch style processing patterns
- End-to-end lakehouse architecture ideas
Blog link : https://medium.com/@pranavsadagopan/building-a-spark-declarative-pipeline-a-modern-financial-data-la...
Looking forward to feedback and discussions ๐
