topic Re: Can we able to create materialized view in databricks using all purpose cluster in Data Engineering

Can we able to create materialized view in databricks using all purpose cluster

Shivaprasad — Sat, 23 May 2026 15:09:12 GMT

I was unable to create materialized view in databricks using all purpose cluster wanted to check do we need serverless cluster to create MV

Re: Can we able to create materialized view in databricks using all purpose cluster

szymon_dybczak — Sat, 23 May 2026 16:53:27 GMT

Hi @Shivaprasad ,

Nope, all purpose cluster are not supported. Standalone materialized views can be created/refreshed either from a Unity Catalog-enabled Pro or Serverless SQL Warehouse, or from a notebook attached to Serverless General Compute.

So we don’t necessarily need a “serverless cluster” specifically, but we do need one of the supported compute options - I.e a SQL Warehouse. An all-purpose cluster is not supported for creating standalone materialized views.

For materialized view created via pipeline you can choose serverless or classic compute:

https://docs.databricks.com/aws/en/ldp/configure-compute

Here is similar thread:

https://community.databricks.com/t5/data-engineering/spark-declarative-pipelines-use-in-all-purpose-compute/td-p/144076

If my answer was helpful, please consider marking it as accepted solution

Re: Can we able to create materialized view in databricks using all purpose cluster

Shivaprasad — Sat, 23 May 2026 20:08:11 GMT

Thanks. we have SQL warehouse cluster and has storage_blob_contributor_role_on_storage_account but I am still getting below error -

Encountered an error with Unity Catalog while setting up the pipeline on cluster 0521-174117-2fdl63ad-v2n.
Ensure that your Unity Catalog configuration is correct, and that required resources (e.g., catalog, schema) exist and are accessible.
Also verify that the cluster has appropriate permissions to access Unity Catalog.

Details: Operation failed: "Forbidden", 403, GET, http://sa14127e1dv0201.dfs.core.windows.net/find?upn=false&beginFrom=0000000000000000000&resource=filesystem&maxResults=5000&directory=unity_catalog/erp_inbound/__unitystorage/schemas/0acf8b40-5cdd-486b-aa72-3e5f507cec3c/tables/1df52346-9e8f-4d90-a379-74347daed332/_delta_log&timeout=90&recursive=false&st=2026-05-21T17:00:12Z&sv=2020-02-10&ske=2026-05-21T19:00:12Z&sig=XXXXX&sktid=ca56a4a5-e300-406a-98ff-7e36a0baac5b&se=2026-05-21T18:41:31Z&sdd=7&skoid=a3f681e5-266f-4a68XXXXXXXXXXXXXXXXXX&spr=https&sks=b&skt=2026-05-21T17:00:12Z&sp=racwdxlm&skv=2025-01-05&sr=d, AuthorizationFailure, , "This request is not authorized to perform this operation. RequestId:b35f3d5a-e01f-0073-0f49-e92e3b000000 Time:2026-05-21T17:41:35.2242794Z"

==========================================================================
workspace : https://adb-1976997951002494.14.azuredatabricks.net/editor/files/370237779897823?o=1976997951002494

catalog and schema has create martialized view create privileges

Re: Can we able to create materialized view in databricks using all purpose cluster

szymon_dybczak — Sun, 24 May 2026 08:08:27 GMT

Hi @Shivaprasad ,

It can be networking issue. Could you tell us your networking setup? Does your storage account has public network access disabled?
Are you using standalone materialized view or you're creating MV via SDP pipeline?

Re: Can we able to create materialized view in databricks using all purpose cluster

Ashwin_DSA — Sun, 24 May 2026 13:55:55 GMT

Hi @Shivaprasad,

You generally should not create a standalone materialised view from an all-purpose cluster. Databricks documents that CREATE MATERIALIZED VIEW is supported from a Pro or Serverless SQL warehouse, or within a pipeline. For standalone materialised views, Databricks also documents that you can create them from a Databricks SQL warehouse or from a notebook running on serverless general compute.

So if you are trying this from an all-purpose cluster, that part is already a problem. A SQL warehouse is the correct route.

The second part is the important one... even when you issue the command from a SQL warehouse, Databricks creates and uses a serverless backing pipeline for the create/refresh path. Databricks explicitly says that create and refresh operations for materialised views are powered by serverless Lakeflow pipelines, and that a serverless pipeline is automatically created for Databricks SQL materialised views.

That is why the error you shared does not appear to be a simple SQL warehouse permission issue. The error is a 403 against the Unity Catalog managed storage path under __unitystorage/.../_delta_log, which strongly suggests that the backing serverless pipeline cannot access the Azure Data Lake Storage location used by Unity Catalog.

A few things you can check...

First of all, make sure you are using a Pro or Serverless SQL warehouse for CREATE MATERIALIZED VIEW, not an all-purpose cluster.

On Azure, check that Unity Catalog storage access is configured using the correct Databricks-managed identity / Access Connector path, and that the required role is assigned to that identity. The Azure Databricks guidance for Unity Catalog storage access calls out assigning the role to the managed identity, not just relying on a user-level permission.

Also, double check the exact Azure RBAC role. The documented role is Storage Blob Data Contributor, which is different from "Storage Blob Contributor".

Lastly, if the storage account is behind firewall rules, Private Link, or a VNet-injected setup, you should also verify that serverless access is configured correctly for that storage account. Databricks documents additional setup for managed identity trusted access and serverless private connectivity.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.