Custom Catalog Not Visible to ADF and Looker in Databricks

[palearnings]

Hi Databricks community,

I’ve created a catalog called azure_data_factory and a database called studytube_azure_mysql_production. Within that database, I have three tables: academies, academy_naming_settings, and users.
See the left side of the screenshot.

However, when I try to connect Databricks to Azure Data Factory, ADF only displays the tables and the database studytube_azure_sql_production under the default catalog.
See the right side of the screenshot.

I’ve already granted the necessary permissions (USE and SELECT).

I’m experiencing the same issue when connecting Databricks to Looker — Looker can only see the tables and the database under the default catalog.

Ideally, I want all data replicated from ADF to be placed in schemas under the azure_data_factory catalog.
Has anyone encountered this issue?

[palearnings]

[szymon_dybczak]

Hi @palearnings ,

To query a table in Unity Catalog you need at minimum following privileges:

- USE CATALOG - on the parent catalog of the table

- USE SCHEMA - on the parent schema of the table

- SELECT on the table itself

Did you set all of those privileges on your ADF service principal?

If you did that then refer to following yt video. It shows how to setup adf dataset to point to specific UC table:

https://youtu.be/saVZdf6cQig?t=187

[Raman_Unifeye]

As mentioned by @szymon_dybczak, it is certainly a 'Grant' issue. Pls try granting the USE permissions to your SP first. Hope that resolves your issue.

[palearnings]

Hi @szymon_dybczak  and @Raman_Unifeye ,
Thanks for answering my questions. I have granted the same permissions for both catalogues:

• USE CATALOG
• USE SCHEMA on the schemas under both catalogues
• SELECT on the tables under both schemas and catalogues

I even granted ALL PRIVILEGES to my service principal and all account users for testing purposes. However, ADF is still unable to see the catalogue and its contents.

When I was checking that my service principal had the same privileges on both catalogues, I realised that the principal name on my default catalogue is _workspace_users_databricks_34xxx.
See the screenshot.

When I tried granting privileges to _workspace_users_databricks_34xxx, I received the following response:

[RequestId=f9efc352-a596-4197-aea1-debf36a1bdb8 ErrorClass=INVALID_PARAMETER_VALUE.UC_SYSTEM_WORKSPACE_GROUP_PERMISSION_UNSUPPORTED] Cannot grant privileges on CATALOG to system generated group _workspace_users_databricks_34xxx

Do you know what I might have missed?

[Coffee77]

It seems you have a group called "_workspace_uusers_databricks_34xxxx". So, it is not allowed to assign permissions to that group. Verify service principal name and change group name in order to not match and spark unexpected issues 🙂

[palearnings]

Hi @Coffee77,

Thank you for your response. The point I’m trying to address is that I would like my ADF to access my custom catalog, azure_data_factory. I’ve already granted my ADF service principal permission to use the catalog, but it doesn’t seem to resolve the issue.

I’m fairly certain that my service principal ID, starting with c38d**, is correct, as I’ve granted this same service principal access to select tables in my default catalog, and that works without issues.

Best regards,

[Coffee77]

Sorry, I didn't catch it! Please, verify you can access to your target catalog with desired service principal by using other consumer, for instance, Databricks CLI (list catalogs, etc.) in order to make sure problem is the same as ADF. If you can access from CLI, then problem is about ADF not taking correct principal. Really strange issue