topic Re: Custom Catalog Not Visible to ADF and Looker in Databricks in Warehousing & Analytics

Custom Catalog Not Visible to ADF and Looker in Databricks

palearnings — Mon, 17 Nov 2025 17:05:57 GMT

Hi Databricks community,

I’ve created a catalog called azure_data_factory and a database called studytube_azure_mysql_production. Within that database, I have three tables: academies, academy_naming_settings, and users.
See the left side of the screenshot.

However, when I try to connect Databricks to Azure Data Factory, ADF only displays the tables and the database studytube_azure_sql_production under the default catalog.
See the right side of the screenshot.

I’ve already granted the necessary permissions (USE and SELECT).

I’m experiencing the same issue when connecting Databricks to Looker — Looker can only see the tables and the database under the default catalog.

Ideally, I want all data replicated from ADF to be placed in schemas under the azure_data_factory catalog.
Has anyone encountered this issue?

Re: Custom Catalog Not Visible to ADF and Looker in Databricks

palearnings — Mon, 17 Nov 2025 17:06:46 GMT

Re: Custom Catalog Not Visible to ADF and Looker in Databricks

szymon_dybczak — Mon, 17 Nov 2025 17:22:18 GMT

Hi @palearnings ,

To query a table in Unity Catalog you need at minimum following privileges:

- USE CATALOG - on the parent catalog of the table

- USE SCHEMA - on the parent schema of the table

- SELECT on the table itself

Did you set all of those privileges on your ADF service principal?

If you did that then refer to following yt video. It shows how to setup adf dataset to point to specific UC table:

https://youtu.be/saVZdf6cQig?t=187

Re: Custom Catalog Not Visible to ADF and Looker in Databricks

Raman_Unifeye — Mon, 17 Nov 2025 17:29:19 GMT

As mentioned by @szymon_dybczak, it is certainly a 'Grant' issue. Pls try granting the USE permissions to your SP first. Hope that resolves your issue.

Re: Custom Catalog Not Visible to ADF and Looker in Databricks

palearnings — Tue, 18 Nov 2025 11:38:17 GMT

Hi @szymon_dybczak and @Raman_Unifeye ,
Thanks for answering my questions. I have granted the same permissions for both catalogues:

USE CATALOG
USE SCHEMA on the schemas under both catalogues
SELECT on the tables under both schemas and catalogues

I even granted ALL PRIVILEGES to my service principal and all account users for testing purposes. However, ADF is still unable to see the catalogue and its contents.

When I was checking that my service principal had the same privileges on both catalogues, I realised that the principal name on my default catalogue is _workspace_users_databricks_34xxx.
See the screenshot.

When I tried granting privileges to _workspace_users_databricks_34xxx, I received the following response:

[RequestId=f9efc352-a596-4197-aea1-debf36a1bdb8 ErrorClass=INVALID_PARAMETER_VALUE.UC_SYSTEM_WORKSPACE_GROUP_PERMISSION_UNSUPPORTED] Cannot grant privileges on CATALOG to system generated group _workspace_users_databricks_34xxx

Do you know what I might have missed?

Re: Custom Catalog Not Visible to ADF and Looker in Databricks

Coffee77 — Tue, 18 Nov 2025 12:08:51 GMT

It seems you have a group called "_workspace_uusers_databricks_34xxxx". So, it is not allowed to assign permissions to that group. Verify service principal name and change group name in order to not match and spark unexpected issues 🙂

Re: Custom Catalog Not Visible to ADF and Looker in Databricks

palearnings — Tue, 18 Nov 2025 12:55:14 GMT

Hi @Coffee77,

Thank you for your response. The point I’m trying to address is that I would like my ADF to access my custom catalog, azure_data_factory. I’ve already granted my ADF service principal permission to use the catalog, but it doesn’t seem to resolve the issue.

I’m fairly certain that my service principal ID, starting with c38d**, is correct, as I’ve granted this same service principal access to select tables in my default catalog, and that works without issues.

Best regards,

Re: Custom Catalog Not Visible to ADF and Looker in Databricks

Coffee77 — Tue, 18 Nov 2025 13:08:42 GMT

Sorry, I didn't catch it! Please, verify you can access to your target catalog with desired service principal by using other consumer, for instance, Databricks CLI (list catalogs, etc.) in order to make sure problem is the same as ADF. If you can access from CLI, then problem is about ADF not taking correct principal. Really strange issue