Dario Schiraldi : How do I build a data pipeline in Databricks?

darioschiraldi9 — Tue, 08 Jul 2025 05:22:21 GMT

Hey everyone,

I am Dario Schiraldi, working on building a data pipeline in Databricks and would love to get some feedback and suggestions from the community. I want to build a scalable and efficient pipeline that can handle large datasets and possibly integrate with cloud storage like AWS S3 or Azure Blob.

Looking forward to hearing your thoughts and suggestions!

Regards

Dario Schiraldi CEO of Travel Works

Re: Dario Schiraldi : How do I build a data pipeline in Databricks?

ilir_nuredini — Tue, 08 Jul 2025 09:00:49 GMT

Hello @darioschiraldi9 ,

Happy to hear that that you are exploring Databricks for you work. Here you may find a very detailed and good example on how you can build scalable data pipeline using DLT and with the flexibility of Spark Streaming and a sophisticated configuration-driven approach. :

https://community.databricks.com/t5/technical-blog/lakeflow-config-driven-framework-a-guide-to-building-scalable/ba-p/116841

And in this link, if though it is old, you may find some very useful information on architectural level :
https://www.youtube.com/watch?v=9sBdD1G34Mg

Hope that helps. Also if you give more information on your project in terms of technology details we can compile a better suggestion. Thank you!

Best,Ilir

topic Dario Schiraldi : How do I build a data pipeline in Databricks? in Data Engineering

Dario Schiraldi : How do I build a data pipeline in Databricks?

Re: Dario Schiraldi : How do I build a data pipeline in Databricks?