Current Azure Managed Identity capabilities 2024?

[zsucic1]

Hello everyone, 

I have a few questions about MI capabilites: 

1. Is it possible to define a managed identity for Azure Databricks Service resource and use it for e.g.: 
   1. Writing to Azure SQL Server database 
   2. Authenticating to Azure Devops in order to download a repo 
2. Is it possible to define a managed identity for Azure App Service resource and use it to download MLFLOW models from Databricks (Workspace/unity catalog based) model registry, using  

azure-identity Managed Identity library in python code to authenticate? If yes, are you aware of any additional steps to take, other than: 

1. Turning on managed identity on App Service 
2. Adding that managed identity to Databricks (as a service principal, which is required by Databricks) 

When I did those steps, I got 403 authentication error, is it likely to simply be a mistake on our part? 

1. Is it possible to use a service principal to download a Azure DevOps repo from a Databricks job, by linking a Databricks Git credentials entry to it, as opposed to a mere user like user@gmail.com? 

Any answers, links, tips and comments are greatly appreciated. 

Thanks in advance!  

[Kaniz · Accepted solution]

Hi @zsucic1 , Let’s dive into your questions about managed identities (MIs) and their capabilities in Azure Databricks and Azure App Service.

1. Managed Identity for Azure Databricks Service Resource:

   • Yes, you can define a managed identity for an Azure Databricks Service resource. This managed identity allows programmatic calls to Azure Databricks account and workspace operations using the identity of the managed service identity (MSI).
   • It’s particularly useful when working with Azure resources that support managed identities, such as Azure VMs.
   • To set up and use managed identities authentication for Azure Databricks, follow these steps:
     1. Enable Managed Identity: Turn on managed identity for your Azure Databricks Service resource.
     2. Configure Environment Variables: Set the necessary environment variables or fields on a properly supported Azure VM. These include the Azure Databricks host, account ID, tenant ID, client ID, and Azure resource ID.
     3. Integrate with Your Code: Depending on the tool or SDK you’re using, integrate the managed identity within your code. For example, you can use environment variables or fields specific to your chosen authentication type.
   • Note that managed identities for Azure resources are different from Microsoft Entra ID service princ...¹.

2. Managed Identity for Azure App Service Resource:

   • Yes, you can create a managed identity for an Azure App Service resource. Managed identities allow your app to obtain tokens for Azure resources protected by Microsoft Entra ID, such as Azure SQL Database, Azure Key Vault, and Azure Storage.
   • There are two types of identities:
     • System-assigned identity: Tied to your application and deleted if your app is deleted.
     • User-assigned identity: A standalone Azure resource that can be assigned to your app.
   • To set up a managed identity for App Service:
     1. System-Assigned Identity:
        • In the Azure portal, navigate to your app’s settings, select “Identity,” and switch the status to “On.”
     2. User-Assigned Identity:
        • Create a user-assigned identity resource.
        • Add the identity’s resource identifier to your app config.
     3. Configure Target Resources: Ensure that downstream resources (e.g., Azure Key Vault) allow access from your app or function.
   • Managed identities simplify authentication and eliminate the need to provision or rotate secrets ².

3. Using Service Principal to Download Azure DevOps Repo from Databricks:

   • Yes, you can use a service principal for this purpose.
   • Instead of linking a Databricks Git credentials entry to a user account, associate it with a service principal.
   • This approach provides better security and avoids relying on individual user accounts like user@gmail.com.
   • Ensure that the service principal has the necessary permissions to access the Azure DevOps repo.
   • If you encountered a 403 authentication error, double-check the permissions and ensure that the service principal is correctly configured.

Remember that managed identities enhance security and simplify authentication across Azure services. Feel free to explore the provided links for more detailed instructions and best practices! 🚀🔐

[Kaniz]

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the response and accept it as a solution?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

[zsucic1]

Hi Kaniz,

Thank you immensely for this thorough response!
You really helped us a lot
and enabled us to start resolving this problem strategically.
May I just ask you a few follow ups:

1. Regarding your 2nd point, do you know specifically if it is possible
to download Mlflow models as a Azure Managed Identity? If it is then
we made a mistake somewhere and we can systematically try to resolve it.

2. Regarding your 3rd point, thank you,
service principal is certainly an improvement over user account,
but do you know if this could be further improved by downloading
Azure DevOps Repo as a "Azure Databricks Service Managed identity",
instead of as a service principal. A corrollary question would be:
is it possible to run Databricks jobs as a
"Azure Databricks Service Managed identity",
instead of as a service principal.

[Kaniz]

Hi @zsucic1, You can deploy MLflow models as Azure web services and leverage Azure Machine Learning’s model management capabilities.

Let’s explore how you can achieve this:

1. Prerequisites:

   • Ensure you have a trained machine learning model. If not, find a suitable notebook example in the repository and follow the instructions.
   • Set up the MLflow Tracking URI to connect with Azure Machine Learning.
   • Install the azureml-mlflow package, which automatically includes azureml-core from the Azure Machine Learning Python SDK.
   • Verify that you have the necessary access permissions for your MLflow operations within your workspace.

2. Deployment Options:

   • Azure Container Instance (ACI): Ideal for quick dev-test deployments. No specific deployment configuration is required; the service defaults to ACI deployment when no config is provided.
   • Azure Kubernetes Service (AKS): Recommended for scalable production deployments.
   • Azure Machine Learning Endpoints (v2): These provide an improved, simpler deployment experience, supporting both real-time and batch inference scenarios. Endpoints unify model deployments across compute types¹.

3. Deploying to ACI:

   • Ensure your MLflow model is set up with the MLflow Tracking URI to connect with Azure Machine Learning.
   • Deploy your model to an Azure Machine Learning web service. The service will default to ACI deployment when no specific configuration is provided¹.

4. Additional Resources:

   • Explore the integration between MLflow and Azure Machine Learning for more functionality integrations.
   • If you’re an administrator interested in monitoring resource usage and events from Azure Machine Lea...¹.

Remember, this combination of MLflow and Azure Machine Learning allows you to manage the entire lifecycle of your machine learning experiments, from training to deployment, all within your workspace. If you encounter any issues, systematically review your setup to identify any mistakes and resolve them. 🚀🔍

[Kaniz]

Hi @zsucic1, For your second question, Let’s delve into the world of Azure Databricks and explore how managed identities can enhance security and streamline authentication.

1. Azure Managed Identities Authentication for Azure Databricks:

   • Managed identities (formerly known as Managed Service Identities or MSIs) provide a secure way to authenticate with Azure resources. They eliminate the need to manage credentials explicitly.
   • When working with Azure Databricks, you can use a managed identity to obtain Azure Active Directory (Azure AD) tokens without handling any credentials directly.
   • These managed identities are automatically managed by Azure for applications connecting to resources...¹.
   • To set up and use Azure managed identities authentication for Azure Databricks, follow these steps:
     • Create a user-assigned managed identity (recommended).
     • Assign the managed identity to your Azure Databricks account and workspace.
     • Log in to an Azure virtual machine (VM) that supports managed identities.
     • Install the Databricks CLI on the VM and configure it for Azure managed identities authentication.
     • Use the Databricks CLI to automate your Azure Databricks account and workspace operations using the ...¹.

2. Benefits of Using Managed Identities:

   • Credential-free: Managed identities eliminate the need to handle secrets or service principal credentials explicitly.
   • Automated token retrieval: The managed identity obtains Azure AD tokens seamlessly.
   • Enhanced security: No need to store or manage sensitive credentials.
   • Supported resources: Azure managed identities authentication works between properly configured resou...¹.

3. Corollary Answer:

   • Yes, you can run Databricks jobs using an Azure Databricks Service Managed identity instead of a service principal.
   • By leveraging managed identities, you enhance security and simplify authentication for your Databricks workloads.

Remember that managed identities for Azure resources are distinct from Microsoft Entra ID (formerly Azure AD) service principals, which Azure Databricks also supports for authentication. Choose the approach that best aligns with your security requirements and operational ease¹.

Feel free to explore this powerful authentication mechanism and elevate your Azure Databricks experience! 🚀🔐

[zsucic1]

Kaniz, thank you very much, you are the best! I will get to work implementing your advice 🙂