- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thursday
We are experiencing an intermittent issue when reading Unity Catalog materialized views from downstream transformation notebooks.
Environment
- Azure Databricks
- Unity Catalog enabled
- Materialized views and streaming tables are created by a Lakeflow (formerly DLT) pipeline
- Source data is ingested from external storage as Parquet files
- Downstream transformation notebooks run as job tasks on Dedicated (Single User) Compute
- Jobs execute under the same service principal that owns the pipeline and materialized views
- All reads are performed using table/view names (spark.table(...) or SQL)
- No path-based reads are used anywhere in the application code
- Cluster policy uses auto:latest-lts, which currently provisions DBR 18.x runtimes
Issue
Since approximately October 2026, some jobs have started failing intermittently with the following error during the first read of a materialized view:
The reported path corresponds to the internal Unity Catalog managed storage location that backs the materialized view.
Observed Behavior
- Failure occurs on the first read of a materialized view.
- Reads of regular managed tables and streaming tables continue to work successfully.
- Only materialized-view reads are affected.
- Retrying the task on the same cluster consistently fails.
- Repairing/rerunning the job on a newly created cluster using the same runtime version succeeds.
- No code, permissions, storage configuration, or pipeline changes were made.
Historical Timeline
- Workloads have been running successfully for more than six months.
- Previously ran successfully on:
- DBR 17.3 LTS
- DBR 18.x releases observed prior to the issue
- Serverless compute
- Failures began after clusters started showing a newer DBR 18.3.x maintenance release. Recently my job cluster used 18.3.9 release.
What Has Been Ruled Out
- Concurrent pipeline updates
- Stale or dropped materialized views
- Ownership/permission issues
- External location overlap
- Code regressions
- Data changes
The pipeline was idle during several failures, and the same identities retain ownership and access to the affected objects.
Working Theory
The error appears to originate from Unity Catalog's managed-storage validation (CheckPathAccess) when resolving the underlying storage location of materialized views.
Given that:
- The same code succeeds on serverless compute.
- The same code succeeds on newly provisioned clusters.
- The same code worked for months before the recent runtime update (approx. running without any issues for 6 months)
- The failure specifically targets materialized views and not streaming tables or managed tables.
We suspect there may have been a change in how Dedicated Compute resolves materialized-view storage access, potentially involving:
- Table Credential resolution
- Managed storage path validation
- Unity Catalog authorization behavior
- Materialized-view materialization access paths
Questions
Has anyone observed similar LOCATION_OVERLAP or CheckPathAccess failures when reading Unity Catalog materialized views on recent DBR 18.x runtimes?
Were there any changes in how Dedicated Compute accesses materialized-view materializations compared with Serverless Compute?
Is there a known regression or behavioral change related to Unity Catalog managed-storage validation in recent DBR 18.x releases?
Any insights, known issues, release notes, KB articles, or recommended mitigations would be greatly appreciated
Databricks Data Engineer
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thursday
CheckPathAccess rejects it with LOCATION_OVERLAP. The rejection isn't handled, so the whole query fails. Streaming tables and managed tables resolve with full table metadata and get table-level access, which is why only materialized views are affected.- Known issue? Yes, it's the bug above, and Databricks is working on a fix.
- Dedicated vs serverless: dedicated compute resolves materialized views through a different code path, which is where the bug is. Serverless doesn't take that path.
- Regression? Yes. The change is being rolled out gradually and applies per cluster, which explains both the intermittency and the clean runs on new clusters. The timing matches the 18.3.x maintenance releases, but the runtime itself isn't the cause.
Thanks again for raising it.
Nina
References
Databricks Senior Solutions Engineer
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thursday
Adding a small practical note to Nina's answer, since your policy uses auto:latest-lts.
The quickest way to apply the 17.3 LTS workaround without hardcoding a version key is to switch the policy to auto:prev-lts. It maps to the second-latest LTS runtime, which today is 17.3 LTS (latest LTS is 18):
{
"spark_version": { "type": "fixed", "value": "auto:prev-lts" }
}
Two things to keep in mind:
- Changing the policy affects every cluster that uses it. If only a few jobs hit the error, clone the policy and attach the clone just to those jobs.
- auto:prev-lts moves forward when the next LTS ships, so treat this as temporary and switch back to auto:latest-lts once the fix is out. 17.3 LTS is supported until October 22, 2028, so there's no rush from that side.
Docs:
Compute policy reference (special spark_version values): https://docs.databricks.com/aws/admin/clusters/policy-definition
Runtime versions and support dates: https://docs.databricks.com/aws/en/release-notes/runtime
Principal Data Architect & AI Strategy — CI&T
thomazn@ciandt.com
linkedin.com/in/thomaz-antonio-rossito-neto | Databricks Solutions Architect Champion