Hello @darioschiraldi9 ,

Happy to hear that that you are exploring Databricks for you work. Here you may find a very detailed and good example on how you can build scalable data pipeline using DLT and  with the flexibility of Spark Streaming and a sophisticated configuration-driven approach. :

https://community.databricks.com/t5/technical-blog/lakeflow-config-driven-framework-a-guide-to-build...

And in this link, if though it is old, you may find some very useful information on architectural level :
https://www.youtube.com/watch?v=9sBdD1G34Mg

Hope that helps. Also if you give more information on your project in terms of technology details we can compile a better suggestion. Thank you!

Best,Ilir