topic Re: Possible false positive warning on DLT pipeline in Data Engineering

Possible false positive warning on DLT pipeline

ipreston — Wed, 17 Jan 2024 17:27:40 GMT

I have a DLT pipeline script that starts by extracting metadata on the tables it should generate from a delta table. Each record returned from the table should be a dlt table to generate, so I use .collect() to turn each row into a list and then iterate on calling my DLT pipeline logic on it. I don't use .collect() in any dlt functions or functions that have a dlt decorator on them. When I run the pipeline I get a warning

Notebook:/<my path here>/dlt_meta_pipeline used `DataFrame.collect` function that will be deprecated soon. Please fix the notebook.

Delta Live Tables job fails when using collect() - Databricks

based on the above post I think I should only be seeing this if I'm using collect to return results that I want to instantiate as a dlt managed table. Is this warning in error or do I actually have to change something?

Re: Possible false positive warning on DLT pipeline

jose_gonzalez — Wed, 17 Jan 2024 21:53:42 GMT

Hi @ipreston,

Could you share the DBR version please? also share the full stack message.

Re: Possible false positive warning on DLT pipeline

ipreston — Wed, 17 Jan 2024 21:57:51 GMT

It's DLT Stable, so whatever DBR that's using under the hood. Here's the JSON of the warning with UIDs etc redacted

{
"id": "redacted",
"sequence": {
"data_plane_id": {
"instance": "execution",
"seq_no": redacted
},
"control_plane_seq_no": redacted
},
"origin": {
"cloud": "Azure",
"region": "canadacentral",
"org_id": redacted,
"pipeline_id": "redacted",
"pipeline_type": "WORKSPACE",
"pipeline_name": "redacted",
"cluster_id": "redacted",
"update_id": "redacted",
"request_id": "redacted",
"uc_resource_id": "redacted"
},
"timestamp": "2024-01-17T15:15:47.639Z",
"message": "Notebook:/redacted/dlt_meta_pipeline used `DataFrame.collect` function that will be deprecated soon. Please fix the notebook.",
"level": "WARN",
"details": {
"unsupported_operation": {
"operation": "COLLECT_TO_DRIVER"
}
},
"event_type": "unsupported_operation",
"maturity_level": "STABLE"
}

Re: Possible false positive warning on DLT pipeline

ipreston — Wed, 17 Jan 2024 22:02:09 GMT

The code is a fork of dlt-meta. Here's where it does the same operation: https://github.com/databrickslabs/dlt-meta/blob/2a93dd9ae42dfdb167b73629bd1acc8a256e7a49/src/dataflow_spec.py#L150

Re: Possible false positive warning on DLT pipeline

jose_gonzalez — Thu, 18 Jan 2024 00:14:14 GMT

Thank you for sharing more details. In this case, this is a warning message level ("level": "WARN",), so it should be fine as long as you are not getting FATAL error level.

Re: Possible false positive warning on DLT pipeline

ipreston — Mon, 22 Jan 2024 16:01:00 GMT

Thanks for the reply. Based on that response though, it seems like the warning itself is a bug in the DLT implementation. Per the docs "However, you can include these functions outside of table or view function definitions because this code is run once during the graph initialization phase." Is there a way to report this issue upstream? I'm not concerned about my code failing as a result of this warning, but I'd like to avoid having false positive alerts in the pipeline as it increases the risk that I'll miss an important warning amidst irrelevant ones like this.

Re: Possible false positive warning on DLT pipeline

Bal — Wed, 08 May 2024 19:00:04 GMT

Are you using Take() or First() in your code? I was using collect and get the same warning, but have since changed to Take() and still get the warning.