topic Re: Issue Reading DLT-Generated Materialized Views on DBR 18.x in Data Engineering

Issue Reading DLT-Generated Materialized Views on DBR 18.x

karuppusamy — Thu, 29 Oct 2026 13:30:28 GMT

We are experiencing an intermittent issue when reading Unity Catalog materialized views from downstream transformation notebooks.

Environment

Azure Databricks
Unity Catalog enabled
Materialized views and streaming tables are created by a Lakeflow (formerly DLT) pipeline
Source data is ingested from external storage as Parquet files
Downstream transformation notebooks run as job tasks on Dedicated (Single User) Compute
Jobs execute under the same service principal that owns the pipeline and materialized views
All reads are performed using table/view names (spark.table(...) or SQL)
No path-based reads are used anywhere in the application code
Cluster policy uses auto:latest-lts, which currently provisions DBR 18.x runtimes

Issue

Since approximately October 2026, some jobs have started failing intermittently with the following error during the first read of a materialized view:

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP

Input path overlaps with managed storage within 'CheckPathAccess' call.

The reported path corresponds to the internal Unity Catalog managed storage location that backs the materialized view.

Observed Behavior

Failure occurs on the first read of a materialized view.
Reads of regular managed tables and streaming tables continue to work successfully.
Only materialized-view reads are affected.
Retrying the task on the same cluster consistently fails.
Repairing/rerunning the job on a newly created cluster using the same runtime version succeeds.
No code, permissions, storage configuration, or pipeline changes were made.

Historical Timeline

Workloads have been running successfully for more than six months.
Previously ran successfully on:
- DBR 17.3 LTS
- DBR 18.x releases observed prior to the issue
- Serverless compute
Failures began after clusters started showing a newer DBR 18.3.x maintenance release. Recently my job cluster used 18.3.9 release.

What Has Been Ruled Out

Concurrent pipeline updates
Stale or dropped materialized views
Ownership/permission issues
External location overlap
Code regressions
Data changes

The pipeline was idle during several failures, and the same identities retain ownership and access to the affected objects.

Working Theory

The error appears to originate from Unity Catalog's managed-storage validation (CheckPathAccess) when resolving the underlying storage location of materialized views.

Given that:

The same code succeeds on serverless compute.
The same code succeeds on newly provisioned clusters.
The same code worked for months before the recent runtime update (approx. running without any issues for 6 months)
The failure specifically targets materialized views and not streaming tables or managed tables.

We suspect there may have been a change in how Dedicated Compute resolves materialized-view storage access, potentially involving:

Table Credential resolution
Managed storage path validation
Unity Catalog authorization behavior
Materialized-view materialization access paths

Questions

Has anyone observed similar LOCATION_OVERLAP or CheckPathAccess failures when reading Unity Catalog materialized views on recent DBR 18.x runtimes?
Were there any changes in how Dedicated Compute accesses materialized-view materializations compared with Serverless Compute?
Is there a known regression or behavioral change related to Unity Catalog managed-storage validation in recent DBR 18.x releases?

Any insights, known issues, release notes, KB articles, or recommended mitigations would be greatly appreciated

Re: Issue Reading DLT-Generated Materialized Views on DBR 18.x

NinaThoni — Thu, 29 Oct 2026 15:22:58 GMT

Hi @karuppusamy,

Thanks for the thorough write-up. Your theory is correct, and nothing on your side caused this.

The bug: Unity Catalog recently added a step during query analysis that works out which storage access each query needs. It's meant to be non-blocking. On dedicated compute, a materialized view's backing table is resolved without its full table metadata. So the new step doesn't treat it as a table. It requests access to the backing table's raw storage path instead. That path sits under managed storage, which Unity Catalog never allows path-based access to, so CheckPathAccess rejects it with LOCATION_OVERLAP. The rejection isn't handled, so the whole query fails. Streaming tables and managed tables resolve with full table metadata and get table-level access, which is why only materialized views are affected.

To your questions:

Known issue? Yes, it's the bug above, and Databricks is working on a fix.
Dedicated vs serverless: dedicated compute resolves materialized views through a different code path, which is where the bug is. Serverless doesn't take that path.
Regression? Yes. The change is being rolled out gradually and applies per cluster, which explains both the intermittency and the clean runs on new clusters. The timing matches the 18.3.x maintenance releases, but the runtime itself isn't the cause.

Mitigations until the fix is out: pin the affected job clusters to DBR 17.3 LTS, or move those tasks to serverless. If it's blocking production, open a support ticket with your workspace ID, a failing run URL and the error's request ID. Support can disable the change for your workspace. I'm also happy to take care of this for you if you send me those details.

Thanks again for raising it.
Nina

References

Re: Issue Reading DLT-Generated Materialized Views on DBR 18.x

ThomazNeto — Thu, 29 Oct 2026 16:10:32 GMT

Adding a small practical note to Nina's answer, since your policy uses auto:latest-lts.

The quickest way to apply the 17.3 LTS workaround without hardcoding a version key is to switch the policy to auto:prev-lts. It maps to the second-latest LTS runtime, which today is 17.3 LTS (latest LTS is 18):

{
"spark_version": { "type": "fixed", "value": "auto:prev-lts" }
}

Two things to keep in mind:

- Changing the policy affects every cluster that uses it. If only a few jobs hit the error, clone the policy and attach the clone just to those jobs.
- auto:prev-lts moves forward when the next LTS ships, so treat this as temporary and switch back to auto:latest-lts once the fix is out. 17.3 LTS is supported until October 22, 2028, so there's no rush from that side.

Docs:
Compute policy reference (special spark_version values): https://docs.databricks.com/aws/admin/clusters/policy-definition
Runtime versions and support dates: https://docs.databricks.com/aws/en/release-notes/runtime