topic Re: DLT Apply Changes Tables corrupt in Data Engineering

DLT Apply Changes Tables corrupt

MatthewMills — Mon, 20 Jan 2025 11:56:25 GMT

Got a weird DLT error.

Test harness using the new(ish) 'Apply Changes from Snapshot' Functionality and DLT Serverless (Current Channel). Azure Aus East Region.

Has been working for several months without issue - but within the last week these DLT tables have started failing with an error:

Your request failed with status FAILED: [BAD_REQUEST] Table 'mycatalog.myschema.mytable' is invalid because one of the underlying resources does not exist. Table '__databricks_internal.__dlt_materialization_schema_##################.__materialization_mat_mytable_1' does not exist.

1) DLT does not need to be running for the tables to stop working. Within a few hours of the pipeline running the tables in the job all stop working with the above error.

2) If the broken tables are manually deleted via the UC browser - DLT undrops the tables and recreates them correctly. A few hours later they break.

3) This happens across multiple environments within the tenancy.

Re: DLT Apply Changes Tables corrupt

Pelle123 — Thu, 23 Jan 2025 10:20:36 GMT

I'm experiencing the same issue.

Re: DLT Apply Changes Tables corrupt

mjbobak — Mon, 27 Jan 2025 21:51:56 GMT

We have the same error. It does not seem to be related to the Current or Preview runtimes in the DLT settings.

region: Azure East US 2

For us, the pipeline completes successfully but the corresponding connection to the underlying data referenced from the catalog is missing. Further, this appears to be for apply_changes_in_snapshot() API into a streaming table but tables and data appear fine for standard streaming tables and materialized views.

Re: DLT Apply Changes Tables corrupt

Lakshay — Tue, 28 Jan 2025 05:29:20 GMT

We have an open ticket on this issue. The issue is caused by the maintenance pipeline renaming the backing table. We expect the fix to be rolled out soon for this issue.