Lakeflow Connect & Community Connectors - 403 Errors + What is the compute?

[ChristianRRL]

Hi there,

I am regularly getting 403 errors with Lakeflow Connect or Community Connectors accessing our UC catalog/schema. As far as I'm seeing, I believe that we have the proper role on our Azure storage account (Storage Blob Data Contributor), and the correct grants (USE CATALOG, USE SCHEMA, CREATE TABLE).

I suspect that the core of our issue is in large part due to the compute that is being used under the hood. It's not leveraging our standard all-purpose compute, and I'm not sure if it's leveraging either job compute or serverless compute.

Can anyone help me better understand WHAT compute is being used by Lakeflow Connect & Community Connectors? If using job compute, is there any way to modify the "default" in order to make sure it's compatible with our setup?

[ChristianRRL]

+I should have specified, these are the specific errors I'm regularly seeing:

• UNITY_CATALOG_INITIALIZATION_FAILED
• CLOUD_ACCESS_DENIED

[binlogreader · Accepted solution]

@ChristianRRL Most likely the compute behind these connectors is serverless, and that would explain both parts of your question. SharePoint and the community connectors run on serverless compute only as far as I know.

The docs list serverless as a prerequisite for them, and there is no setting to point them at your all-purpose or job compute instead. And the 403s follow from it.

Your storage firewall allows your VNet, which is where your all-purpose clusters run, so those work. Serverless runs in Databricks' environment outside your VNet, so the firewall rejects it before your roles or grants are ever checked. The error text points the same way: "This request is not authorized to perform this operation" is what Azure storage returns for a network block, while a missing role reads "not authorized to perform this operation using this permission" instead. So your Storage Blob Data Contributor role is probably fine.

You can check this quickly by reading any table in that catalog from a serverless SQL warehouse or a serverless notebook. If that fails with the same 403, the connector is not the problem. The fix can a network connectivity configuration (NCC): an account admin creates it in the account console, attaches it to the workspace, and adds private endpoint rules for the storage accounts, which you then approve on the storage side. It needs to cover the account behind your Unity Catalog managed storage and any external location storage the pipeline writes to. One NCC covers all serverless workloads in the workspace, so this fixes these connectors together.