Dario Schiraldi : How do I build a data pipeline in Databricks?

[darioschiraldi9]

Hey everyone,

I am Dario Schiraldi, working on building a data pipeline in Databricks and would love to get some feedback and suggestions from the community. I want to build a scalable and efficient pipeline that can handle large datasets and possibly integrate with cloud storage like AWS S3 or Azure Blob.

Looking forward to hearing your thoughts and suggestions!

Regards

Dario Schiraldi CEO of Travel Works 

[ilir_nuredini · Accepted solution]

Hello @darioschiraldi9 ,

Happy to hear that that you are exploring Databricks for you work. Here you may find a very detailed and good example on how you can build scalable data pipeline using DLT and  with the flexibility of Spark Streaming and a sophisticated configuration-driven approach. :

https://community.databricks.com/t5/technical-blog/lakeflow-config-driven-framework-a-guide-to-build...

And in this link, if though it is old, you may find some very useful information on architectural level :
https://www.youtube.com/watch?v=9sBdD1G34Mg

Hope that helps. Also if you give more information on your project in terms of technology details we can compile a better suggestion. Thank you!

Best,Ilir