Possible false positive warning on DLT pipeline
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-17-2024 09:27 AM
I have a DLT pipeline script that starts by extracting metadata on the tables it should generate from a delta table. Each record returned from the table should be a dlt table to generate, so I use .collect() to turn each row into a list and then iterate on calling my DLT pipeline logic on it. I don't use .collect() in any dlt functions or functions that have a dlt decorator on them. When I run the pipeline I get a warning
Notebook:/<my path here>/dlt_meta_pipeline used `DataFrame.collect` function that will be deprecated soon. Please fix the notebook.
Delta Live Tables job fails when using collect() - Databricks
based on the above post I think I should only be seeing this if I'm using collect to return results that I want to instantiate as a dlt managed table. Is this warning in error or do I actually have to change something?