- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-02-2025 07:19 AM
Yesterday (10/1) starting around 12 PM EST we starting getting the following error in our Lakeflow Declarative Pipelines (LDP) process. We get this in environments where none of our code has changed. I found some info on the serverless compute about the Databricks Runtime version before the error and after.
This is the version on our last successful run: dlt:16.4.9-delta-pipelines-dlt-release-dp-20250918-rc0-commit-015f1e5-image-20d2b81
And this is the version on all new runs getting the error: dlt:16.4.9-delta-pipelines-dlt-release-dp-20250925-rc0-commit-a18ed15-image-7ea1ff2
Also, the LDPs are based off of a python meta-data driven process (similar to DLT-META) that dynamically creates tables (in bronze and silver) using the dlt.table and dlt.create_streaming_table functions. Have tried many different things but it seems debugging LDPs is very challenging since you can't do print and logging statements (at least not that I've seen).