Can you use Managed Identities in databricks besides Unity Catalog

[ReyCMFG]

We are looking to send messages using databricks to an azure service bus topic and would like to connect to the service bus using a managed identity vs a connection string. Is this possible in databricks. The only thing I could find regarding databricks and managed identities was regarding Unity Catalog and connecting to storage accounts. Thanks

[Anonymous]

@Rey Pitt​ : Yes, it is possible to connect to an Azure Service Bus topic in Databricks using a Managed Identity instead of a connection string. Hope the link helps - https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-managed-service-identity

TLDR;

• Enable Managed Identity for your Databricks workspace or cluster
• Enable Managed Identity for your Databricks workspace or cluster
• Install the azure-servicebus package in the Databricks cluster
• Use the Managed Identity to authenticate:

[ReyCMFG]

@Suteja Kanuri​  Thanks for the reply. I guess the piece I am struggling with is

• Enable Managed Identity for your Databricks workspace or cluster

How is this done. The only thing I could find was to create a databricks access connector and it appears this is only used with Unity Catalog for access to service accounts. Associating a managed identity with databricks to be used with other resources is what I cannot figure out. Would really be nice if we could create a system assigned MI for databricks🙂

[Anonymous]

@Rey Pitt​ : please check if below helps you

1. Go to the Azure portal and navigate to the Databricks workspace or cluster that you want to enable Managed Identity for.
2. Click on the "Identity" tab on the left-hand side of the screen.
3. Select "System-assigned" or "User-assigned" Managed Identity, depending on your requirements.
4. If you choose "System-assigned" Managed Identity, toggle the "Status" button to "On". This will create a new Managed Identity in your Azure subscription and assign it to your Databricks workspace or cluster.
5. If you choose "User-assigned" Managed Identity, click on the "Add" button and select the existing User-assigned Managed Identity from the list.
6. Once you have enabled Managed Identity for your Databricks workspace or cluster, you can use it to authenticate your Databricks cluster with other Azure services like Azure Data Lake Storage Gen2, Azure Key Vault, and more.

Note: Make sure that you have the necessary permissions to enable Managed Identity for your Databricks workspace or cluster.

[JPricks]

Thanks for the description, we have tried this solution, unfortunately the mentioned "Identity tab" is missing in our case. Is there a planned update for this to make managed identities available?

[albros]

I @JPricks  have you finally found the identity tab/configuration?

Thanks

[Anonymous]

Hi @Rey Pitt​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

[JPricks]

please take a look at my comment above

[RaocoSolutions]

MSI No, But you can use userAssignedManaged identity which is assigned to databaricks in the managed workspace. Link for your reference 
https://medium.com/@manoj__kumar/leveraging-azure-msi-authentication-in-databricks-notebooks-to-conn...

[johschmidt42]

@RaocoSolutions This works. However, when I asked the Databricks Team, they did not recommed it for production use cases.

[Yetii]

To be honest, the Databricks team sucks. They should add a simple Identity tab under the Databricks workspace resource as it is done for other Microsoft services. They don't think about making product easier to maintain and configure, they look from "developers" perspective and force Unity Catalog usage. It is a nice extension but in many cases, you don't need them and it only overcomplicates infrastructure deliver, when you start considering high-level of security. That is why whenever we figured out that Databricks makes our infrastructure and its security too complicated, we force the team to use HDInsight, which is better integrated with all Azure platforms and its services. E.g. when we figured out that Databricks not support simple add User-assigned Managed Identity through standard ARM, CLI or AZ API we ask data engineering team to left Databricks due to incompatibility with Azure Cloud services, especially security. No one will change company procedures because some teams want a nice shiny tool. I'm sorry this is how enterprise bussines loks like, we don't care about your features if you can't ensure compatibility with cloud we are using on all levels.