Accessing Databricks Delta table in ADF using system-defined managed identity

Databricks36 — Thu, 27 Feb 2025 17:31:40 GMT

I am using Lookup activity in ADF which will read the delta table values from databricks. Currently using the system-defined managed identity of the ADF to connect Databricks delta table. I am unable to see my unity catalog database names in the lookup activity configuration even though "Test Connectivity" is successful. Below are the steps implemented but still no luck. Please let me know what i am missing here.

1) Unity catalog is enabled for the workspace

2) Contributor access is given to ADF managed identity in Databricks

3) In Databricks => Catalog => select catalog => permission tab => Given full access to managed identity

Re: Accessing Databricks Delta table in ADF using system-defined managed identity

mark_ott — Wed, 05 Nov 2025 12:43:57 GMT

You are experiencing an issue in Azure Data Factory (ADF) where the Lookup activity does not show your Unity Catalog databases in the configuration dropdown, even though connectivity from ADF to Databricks is successful and you have followed all recommended steps for permissions.

Key Details and Cause

The most common reasons for this problem are:

Catalog visibility: While you may have granted "full access" to your managed identity in the Catalog Permissions, the access settings for Unity Catalog are more granular than legacy Databricks tables. ADF integration with Databricks via Unity Catalog requires explicit permissions not just at the catalog level, but also at the schema (database) and table levels.
Default Catalog Selection: If the workspace is assigned to use a default catalog, but the connection string or configuration in ADF is not specifying the catalog explicitly, the database list may not populate as expected.

Required Actions and Troubleshooting

1. Check Schema and Table Permissions

You need to ensure the managed identity used by ADF has both "USE SCHEMA" and "SELECT" permissions on the relevant schema (database) and tables within Unity Catalog, not just "USE CATALOG".

2. Check Catalog Connection in ADF

In your ADF Lookup activity, confirm that the linked service for Databricks points to the correct catalog in its configuration. Sometimes, Unity Catalog objects won’t appear unless the catalog and schema are explicitly set in the connection or query parameters.

3. Refresh ADF Dataset Metadata

After updating permissions, refresh or recreate your dataset and linked service in ADF to ensure cached metadata doesn’t hide recent changes.

4. Confirm Managed Identity Principal Name

Ensure you’re granting access to the exact ADF managed identity object (the principal name must exactly match what is shown in Azure portal under your ADF’s managed identity details).

5. Unity Catalog Support in ADF

Confirm you are using the latest version of the ADF Databricks connector. Earlier versions may not fully support Unity Catalog browsing features.

Example Permission Commands

In Databricks SQL, you can grant permissions like this:

sql

GRANT USE CATALOG ON CATALOG <catalog_name> TO '<adf-managed-identity>';
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO '<adf-managed-identity>';
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO '<adf-managed-identity>';

Summary Table

Permission Level	Command Example (SQL)	Must be Set?
Catalog	GRANT USE CATALOG ...	Yes
Schema (Database)	GRANT USE SCHEMA ...	Yes
Table	GRANT SELECT ...	Yes

Next Steps

Double-check and set explicit permissions at all three levels: catalog, schema, and table.
Specify catalog/schema if needed in ADF linked service and datasets.
Refresh/recreate affected ADF datasets after permission changes.

This comprehensive approach should allow your ADF Lookup activity to see and access Unity Catalog databases and tables in Databricks. If it remains unresolved, verify compatibility and supported features for your ADF connector version.

topic Re: Accessing Databricks Delta table in ADF using system-defined managed identity in Data Engineering