- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-07-2025 10:22 PM
Hey everyone,
I am Dario Schiraldi, working on building a data pipeline in Databricks and would love to get some feedback and suggestions from the community. I want to build a scalable and efficient pipeline that can handle large datasets and possibly integrate with cloud storage like AWS S3 or Azure Blob.
Looking forward to hearing your thoughts and suggestions!
Regards
Dario Schiraldi CEO of Travel Works
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-08-2025 02:00 AM
Hello @darioschiraldi9 ,
Happy to hear that that you are exploring Databricks for you work. Here you may find a very detailed and good example on how you can build scalable data pipeline using DLT and with the flexibility of Spark Streaming and a sophisticated configuration-driven approach. :
https://community.databricks.com/t5/technical-blog/lakeflow-config-driven-framework-a-guide-to-build...
And in this link, if though it is old, you may find some very useful information on architectural level :
https://www.youtube.com/watch?v=9sBdD1G34Mg
Hope that helps. Also if you give more information on your project in terms of technology details we can compile a better suggestion. Thank you!
Best,Ilir
