Issue Reading DLT-Generated Materialized Views on DBR 18.x

[karuppusamy]

We are experiencing an intermittent issue when reading Unity Catalog materialized views from downstream transformation notebooks.

Environment

• Azure Databricks
• Unity Catalog enabled
• Materialized views and streaming tables are created by a Lakeflow (formerly DLT) pipeline
• Source data is ingested from external storage as Parquet files
• Downstream transformation notebooks run as job tasks on Dedicated (Single User) Compute
• Jobs execute under the same service principal that owns the pipeline and materialized views
• All reads are performed using table/view names (spark.table(...) or SQL)
• No path-based reads are used anywhere in the application code
• Cluster policy uses auto:latest-lts, which currently provisions DBR 18.x runtimes

Issue

Since approximately October 2026, some jobs have started failing intermittently with the following error during the first read of a materialized view:

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP

Input path overlaps with managed storage within 'CheckPathAccess' call.

The reported path corresponds to the internal Unity Catalog managed storage location that backs the materialized view.

Observed Behavior

• Failure occurs on the first read of a materialized view.
• Reads of regular managed tables and streaming tables continue to work successfully.
• Only materialized-view reads are affected.
• Retrying the task on the same cluster consistently fails.
• Repairing/rerunning the job on a newly created cluster using the same runtime version succeeds.
• No code, permissions, storage configuration, or pipeline changes were made.

Historical Timeline

• Workloads have been running successfully for more than six months.
• Previously ran successfully on:
  • DBR 17.3 LTS
  • DBR 18.x releases observed prior to the issue
  • Serverless compute
• Failures began after clusters started showing a newer DBR 18.3.x maintenance release. Recently my job cluster used 18.3.9 release. 

What Has Been Ruled Out

• Concurrent pipeline updates
• Stale or dropped materialized views
• Ownership/permission issues
• External location overlap
• Code regressions
• Data changes

The pipeline was idle during several failures, and the same identities retain ownership and access to the affected objects.

Working Theory

The error appears to originate from Unity Catalog's managed-storage validation (CheckPathAccess) when resolving the underlying storage location of materialized views.

Given that:

• The same code succeeds on serverless compute.
• The same code succeeds on newly provisioned clusters.
• The same code worked for months before the recent runtime update (approx. running without any issues for 6 months)
• The failure specifically targets materialized views and not streaming tables or managed tables.

We suspect there may have been a change in how Dedicated Compute resolves materialized-view storage access, potentially involving:

• Table Credential resolution
• Managed storage path validation
• Unity Catalog authorization behavior
• Materialized-view materialization access paths

Questions

1. Has anyone observed similar LOCATION_OVERLAP or CheckPathAccess failures when reading Unity Catalog materialized views on recent DBR 18.x runtimes?

2. Were there any changes in how Dedicated Compute accesses materialized-view materializations compared with Serverless Compute?

3. Is there a known regression or behavioral change related to Unity Catalog managed-storage validation in recent DBR 18.x releases?

Any insights, known issues, release notes, KB articles, or recommended mitigations would be greatly appreciated

[NinaThoni · Accepted solution]

Hi @karuppusamy,

Thanks for the thorough write-up. Your theory is correct, and nothing on your side caused this.

The bug: Unity Catalog recently added a step during query analysis that works out which storage access each query needs. It's meant to be non-blocking. On dedicated compute, a materialized view's backing table is resolved without its full table metadata. So the new step doesn't treat it as a table. It requests access to the backing table's raw storage path instead. That path sits under managed storage, which Unity Catalog never allows path-based access to, so CheckPathAccess rejects it with LOCATION_OVERLAP. The rejection isn't handled, so the whole query fails. Streaming tables and managed tables resolve with full table metadata and get table-level access, which is why only materialized views are affected.

To your questions:

1. Known issue? Yes, it's the bug above, and Databricks is working on a fix.
2. Dedicated vs serverless: dedicated compute resolves materialized views through a different code path, which is where the bug is. Serverless doesn't take that path.
3. Regression? Yes. The change is being rolled out gradually and applies per cluster, which explains both the intermittency and the clean runs on new clusters. The timing matches the 18.3.x maintenance releases, but the runtime itself isn't the cause.

Mitigations until the fix is out: pin the affected job clusters to DBR 17.3 LTS, or move those tasks to serverless. If it's blocking production, open a support ticket with your workspace ID, a failing run URL and the error's request ID. Support can disable the change for your workspace. I'm also happy to take care of this for you if you send me those details.

Thanks again for raising it.
Nina

References

• Requirements for querying materialized views and streaming tables
• Databricks Support

[ThomazNeto · Accepted solution]

Adding a small practical note to Nina's answer, since your policy uses auto:latest-lts.

The quickest way to apply the 17.3 LTS workaround without hardcoding a version key is to switch the policy to auto:prev-lts. It maps to the second-latest LTS runtime, which today is 17.3 LTS (latest LTS is 18):

{
"spark_version": { "type": "fixed", "value": "auto:prev-lts" }
}

Two things to keep in mind:

- Changing the policy affects every cluster that uses it. If only a few jobs hit the error, clone the policy and attach the clone just to those jobs.
- auto:prev-lts moves forward when the next LTS ships, so treat this as temporary and switch back to auto:latest-lts once the fix is out. 17.3 LTS is supported until October 22, 2028, so there's no rush from that side.

Docs:
Compute policy reference (special spark_version values): https://docs.databricks.com/aws/admin/clusters/policy-definition
Runtime versions and support dates: https://docs.databricks.com/aws/en/release-notes/runtime