cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Can you use Managed Identities in databricks besides Unity Catalog

ReyCMFG
New Contributor II

We are looking to send messages using databricks to an azure service bus topic and would like to connect to the service bus using a managed identity vs a connection string. Is this possible in databricks. The only thing I could find regarding databricks and managed identities was regarding Unity Catalog and connecting to storage accounts. Thanks

10 REPLIES 10

Anonymous
Not applicable

@Rey Pitt​ : Yes, it is possible to connect to an Azure Service Bus topic in Databricks using a Managed Identity instead of a connection string. Hope the link helps - https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-managed-service-identity

TLDR;

  • Enable Managed Identity for your Databricks workspace or cluster
  • Enable Managed Identity for your Databricks workspace or cluster
  • Install the azure-servicebus package in the Databricks cluster
  • Use the Managed Identity to authenticate:

ReyCMFG
New Contributor II

@Suteja Kanuri​  Thanks for the reply. I guess the piece I am struggling with is

  • Enable Managed Identity for your Databricks workspace or cluster

How is this done. The only thing I could find was to create a databricks access connector and it appears this is only used with Unity Catalog for access to service accounts. Associating a managed identity with databricks to be used with other resources is what I cannot figure out. Would really be nice if we could create a system assigned MI for databricks🙂

Anonymous
Not applicable

@Rey Pitt​ : please check if below helps you

  1. Go to the Azure portal and navigate to the Databricks workspace or cluster that you want to enable Managed Identity for.
  2. Click on the "Identity" tab on the left-hand side of the screen.
  3. Select "System-assigned" or "User-assigned" Managed Identity, depending on your requirements.
  4. If you choose "System-assigned" Managed Identity, toggle the "Status" button to "On". This will create a new Managed Identity in your Azure subscription and assign it to your Databricks workspace or cluster.
  5. If you choose "User-assigned" Managed Identity, click on the "Add" button and select the existing User-assigned Managed Identity from the list.
  6. Once you have enabled Managed Identity for your Databricks workspace or cluster, you can use it to authenticate your Databricks cluster with other Azure services like Azure Data Lake Storage Gen2, Azure Key Vault, and more.

Note: Make sure that you have the necessary permissions to enable Managed Identity for your Databricks workspace or cluster.

JPricks
New Contributor III

Thanks for the description, we have tried this solution, unfortunately the mentioned "Identity tab" is missing in our case. Is there a planned update for this to make managed identities available?

albros
New Contributor II

@JPricks  have you finally found the identity tab/configuration?

Thanks

Anonymous
Not applicable

Hi @Rey Pitt​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

JPricks
New Contributor III

please take a look at my comment above

RaocoSolutions
New Contributor II

MSI No, But you can use userAssignedManaged identity which is assigned to databaricks in the managed workspace. Link for your reference 
https://medium.com/@manoj__kumar/leveraging-azure-msi-authentication-in-databricks-notebooks-to-conn...

 

@RaocoSolutions This works. However, when I asked the Databricks Team, they did not recommed it for production use cases.

Yetii
New Contributor II

To be honest, the Databricks team sucks. They should add a simple Identity tab under the Databricks workspace resource as it is done for other Microsoft services. They don't think about making product easier to maintain and configure, they look from "developers" perspective and force Unity Catalog usage. It is a nice extension but in many cases, you don't need them and it only overcomplicates infrastructure deliver, when you start considering high-level of security. That is why whenever we figured out that Databricks makes our infrastructure and its security too complicated, we force the team to use HDInsight, which is better integrated with all Azure platforms and its services. E.g. when we figured out that Databricks not support simple add User-assigned Managed Identity through standard ARM, CLI or AZ API we ask data engineering team to left Databricks due to incompatibility with Azure Cloud services, especially security. No one will change company procedures because some teams want a nice shiny tool. I'm sorry this is how enterprise bussines loks like, we don't care about your features if you can't ensure compatibility with cloud we are using on all levels.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!