Check the permissions side first. A 403 usually points to an authorization issue rather than the cluster itself. Make sure the pipeline’s service principal/user has the required Unity Catalog privileges on the target catalog, schema, and tables, and also verify the storage credentials/external locations if you’re using them. If those look correct, then move on to checking network rules or workspace connectivity for the ingestion cluster. The pipeline cluster is managed by Lakeflow, so cluster-level access is usually less likely unless there are custom policies in place.

Hi! The 403 error you're seeing is almost certainly not coming from SharePoint — it's coming from Azure Storage.

The key clue is in the error detail: access denied to /cas cloud path. The /cas path is Unity Catalog's managed storage location (Catalog Asset Store) on ADLS Gen2. Lakeflow Connect spins up a job cluster that tries to write there, and that cluster's identity is being denied by Azure.

Here are the three things to check, in order of likelihood:

1. Missing IAM role on the UC managed storage account (most common):

The Databricks workspace managed identity needs Storage Blob Data Contributor on the ADLS Gen2 storage account that backs your Unity Catalog metastore. Go to: Azure Portal → Storage Account (the one linked to your UC metastore) → Access Control (IAM) → check that the Databricks managed identity has Storage Blob Data Contributor.

2. Missing Unity Catalog privileges on the destination:

 The user or service principal that created the pipeline needs explicit grants on the target schema

GRANT USE CATALOG ON CATALOG <your_catalog> TO <principal>;
GRANT USE SCHEMA ON SCHEMA <your_catalog>.<your_schema> TO <principal>;
GRANT CREATE TABLE ON SCHEMA <your_catalog>.<your_schema> TO <principal>;

3. Missing Microsoft Graph API permissions on the Azure AD app:

 If the above are fine, check that your app registration has admin consent granted for Sites.Read.All (or Files.ReadWrite.All) under Microsoft Graph API permissions in Azure AD — without admin consent the token is issued but scoped incorrectly.

Why option A is probably it: The error fires during pipeline compute resource setup, before it even touches SharePoint. That timing means the cluster can't reach UC storage — the SharePoint credentials aren't even in play yet at that stage.

Hope this helps!

Hi @GabFernandes & @henry_collins , want to follow up on y'all's input.

A few updates, I checked IAM > Storage Blob Data Contributor, and it looks like it's set appropriately and I'm in that group, so no issues there:

I also checked for possible Missing Unity Catalog privileges, but everything seems in order there as well:

I haven't checked Missing Microsoft Graph API permissions on the Azure AD app at this point, but have low suspicion this is the issue and will leave this for last.

In the meantime, I did want to go back to the error:

Encountered an error with Unity Catalog while setting up the pipeline on cluster ....

• Ensure that your Unity Catalog configuration is correct, and that required resources (e.g., catalog, schema) exist and are accessible.
• Also verify that the cluster has appropriate permissions to access Unity Catalog.

To me this part seems the most likely to be causing some kind of issue. I'm not sure why, but under the hood it seems to be spinning up job compute clusters.. which I'm not sure why it's doing that since I am trying for it to leverage an existing all-purpose compute cluster. And for the job clusters spun up, I have no idea how they are configured and if they work with our UC.

Can I get some assistance to understand better, how/why it's using job compute cluster instead of all-purpose compute? Is there something I should consider doing to ensure that job compute clusters can spin up with the proper level of UC access?

Last thing, maybe related / maybe no, we have a "MANAGED_CATALOG":