shrikant_kulkar
New Contributor III

Hey Ruby, 

Here are few options for you

1. Use Databricks partner connect (fivetran, qlik replicate etc) to send data from on-prem directly into raw / bronze delta tables - there is addtional cost for these software's. 

2. Set up connection between databricks workspace and on prem network using express route - this will allow you to use external connection and foreign catalog to be created directly on your on-prem data sources 

3.  Use custom technique to send data from on-prem data-sources to event hub / kafka streams, then use spark structured streaming / delta live tables to ingest data into your bronze zone. 

4. Replicate your on-prem data sources to cloud - azure offers different replication options to cloud for most of database technologies. You can then use option #2 to connect to databases directly from your databricks environment. 

Each option comes with additional cost, security, restorability, recovery constraints. We have same architecture as yours, we use autoloader to ingest data into bronze tables and also run micro-batch upsets on the stream to update raw tables based on primary keys. For every source table - we have two tables one running as append only and another with merge. its offering us good flexibility in terms of reprocessing, recovery, running multiple tables in parallel in loop using ADF orchestration. Adding new datasources and schema is very easy as its templatized.