SirNicholas1st
New Contributor II

Hey,

Unfortunately I have no experience with the said issue, but could federated queries be an option? Though, its also a possibility that you would face the same authentication issue with them.

Basically you wouldn't ingest the data into Databricks first, but you would fire queries from dbx against the Azure SQL server. 

For reference one way we've used federated queries in medallion architecture is that we "replaced" the ingestion to bronze with them and then build managed tables from the data retrieved to silver and gold. Not actually the best practice but worked really well.

If that sounds something worth trying here are the docs for it https://docs.databricks.com/aws/en/query-federation/sql-server

Also as I don't know your use case, here are also docs for deciding if federation is something that would fit it: https://docs.databricks.com/aws/en/query-federation/database-federation#query-federation-vs-lakeflow...