You are experiencing an issue in Azure Data Factory (ADF) where the Lookup activity does not show your Unity Catalog databases in the configuration dropdown, even though connectivity from ADF to Databricks is successful and you have followed all recommended steps for permissions.
Key Details and Cause
The most common reasons for this problem are:
-
Catalog visibility: While you may have granted "full access" to your managed identity in the Catalog Permissions, the access settings for Unity Catalog are more granular than legacy Databricks tables. ADF integration with Databricks via Unity Catalog requires explicit permissions not just at the catalog level, but also at the schema (database) and table levels.
-
Default Catalog Selection: If the workspace is assigned to use a default catalog, but the connection string or configuration in ADF is not specifying the catalog explicitly, the database list may not populate as expected.
Required Actions and Troubleshooting
1. Check Schema and Table Permissions
-
You need to ensure the managed identity used by ADF has both "USE SCHEMA" and "SELECT" permissions on the relevant schema (database) and tables within Unity Catalog, not just "USE CATALOG".
2. Check Catalog Connection in ADF
-
In your ADF Lookup activity, confirm that the linked service for Databricks points to the correct catalog in its configuration. Sometimes, Unity Catalog objects wonโt appear unless the catalog and schema are explicitly set in the connection or query parameters.
3. Refresh ADF Dataset Metadata
-
After updating permissions, refresh or recreate your dataset and linked service in ADF to ensure cached metadata doesnโt hide recent changes.
4. Confirm Managed Identity Principal Name
-
Ensure youโre granting access to the exact ADF managed identity object (the principal name must exactly match what is shown in Azure portal under your ADFโs managed identity details).
5. Unity Catalog Support in ADF
-
Confirm you are using the latest version of the ADF Databricks connector. Earlier versions may not fully support Unity Catalog browsing features.
Example Permission Commands
In Databricks SQL, you can grant permissions like this:
GRANT USE CATALOG ON CATALOG <catalog_name> TO '<adf-managed-identity>';
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO '<adf-managed-identity>';
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO '<adf-managed-identity>';
Summary Table
| Permission Level |
Command Example (SQL) |
Must be Set? |
| Catalog |
GRANT USE CATALOG ... |
Yes |
| Schema (Database) |
GRANT USE SCHEMA ... |
Yes |
| Table |
GRANT SELECT ... |
Yes |
Next Steps
-
Double-check and set explicit permissions at all three levels: catalog, schema, and table.
-
Specify catalog/schema if needed in ADF linked service and datasets.
-
Refresh/recreate affected ADF datasets after permission changes.
This comprehensive approach should allow your ADF Lookup activity to see and access Unity Catalog databases and tables in Databricks. If it remains unresolved, verify compatibility and supported features for your ADF connector version.
