Hello! My company wants us to only use managed identities for authentication. We have set up Databricks using Terraform, got Unity Catalog and everything, but we're a very small team and I'm struggling to control permissions outside of Unity Catalog. Basically we want to be able to write to an external Azure SQL Database from Databricks notebook using managed identity, we are not allowed to use username+password for SQL server. Only some of us super users should have the permission to write to SQL server, and some of our service principals.

How do we go about doing this? I tried using DefaultAzureCredential in the azure-identity Python library, but it gives me an error message. Maybe I need to set something up on the cluster? Maybe I should just put a bunch of secrets in key vault and do access control on those secrets through secret scopes?

I'm also not even sure if we want to be writing to SQL server. The thing we are doing is ingesting CDF data from a SQL server using ADF, then we want to make the data available in a different SQL database after being processed through our data platform. The final destination is the SQL database which will be queried by a public API. It seems the easiest to write directly from Databricks -> SQL database instead of for example using ADF to ingest CDF data from our delta lake. But with this identities issue, that is increasingly seemingly like a better option.