Hi @GabFernandes & @henry_collins , want to follow up on y'all's input.

A few updates, I checked IAM > Storage Blob Data Contributor, and it looks like it's set appropriately and I'm in that group, so no issues there:

I also checked for possible Missing Unity Catalog privileges, but everything seems in order there as well:

I haven't checked Missing Microsoft Graph API permissions on the Azure AD app at this point, but have low suspicion this is the issue and will leave this for last.

In the meantime, I did want to go back to the error:

Encountered an error with Unity Catalog while setting up the pipeline on cluster ....

• Ensure that your Unity Catalog configuration is correct, and that required resources (e.g., catalog, schema) exist and are accessible.
• Also verify that the cluster has appropriate permissions to access Unity Catalog.

To me this part seems the most likely to be causing some kind of issue. I'm not sure why, but under the hood it seems to be spinning up job compute clusters.. which I'm not sure why it's doing that since I am trying for it to leverage an existing all-purpose compute cluster. And for the job clusters spun up, I have no idea how they are configured and if they work with our UC.

Can I get some assistance to understand better, how/why it's using job compute cluster instead of all-purpose compute? Is there something I should consider doing to ensure that job compute clusters can spin up with the proper level of UC access?

Last thing, maybe related / maybe no, we have a "MANAGED_CATALOG":