topic Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf' in Data Engineering

Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

liquibricks — Thu, 22 Jan 2026 14:27:42 GMT

We have a Lakeflow Spark Declarative Pipeline using the new PySpark Pipelines API. This was working fine until about 7am (Central European) this morning when the pipeline started failing with a PYTHON.NAME_ERROR: name 'kdf' is not defined. Did you mean: 'sdf'?

The code has not changed and nothing in our infrastructure has changed, so i'm not sure why this has suddenly started happening. I've raised an error with Azure (we're on Azure Databricks), but they are slow so i'm shouting into the void here. Any suggestions? 😅

Usage:

from pyspark import pipelines as dp

...

@dp.table(name=f"{mycatalog}.{myschema}.{mytable}")

def create_streaming_table(😞

return (

....

)

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

liquibricks — Thu, 22 Jan 2026 14:30:28 GMT

The Azure region (Norway East) might have something to do with it. An example pipeline works fine in Sweden Central.

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

liquibricks — Thu, 22 Jan 2026 15:09:47 GMT

Another clue: the SQL API seems to work, while a Python-based pipeline still fails with this eror.

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

zkaliszamisza — Thu, 22 Jan 2026 18:46:14 GMT

Hi, we have the same issue - could you share the json with detailed error?
Does it by any chance mention error in line 79?

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

zkaliszamisza — Thu, 22 Jan 2026 18:47:53 GMT

For us it happened in westeurope around the same time

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

liquibricks — Thu, 22 Jan 2026 21:06:39 GMT

I'm not sure if i can share the complete json. For us the json error refers to line 99, which is a blank link in the src file it refers to.

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

liquibricks — Fri, 23 Jan 2026 07:07:02 GMT

Is it still occurring for you today (+24 hours after the start)? It is for us.

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

zkaliszamisza — Fri, 23 Jan 2026 10:07:14 GMT

It is - we have raised a ticket to Databricks

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

liquibricks — Mon, 26 Jan 2026 13:45:00 GMT

It turns out this problem was caused by a package that was pip installed using an init script. This package had for some reason started pulling in pandas 3.x (despite the fact that the package itself had not been updated), and our Databricks contact informed us that DLT does not support Pandas 3 at this time (at least until the next version of the DBR). Once we pinned the pandas version to < 3 then the pipelines started working again.

No idea why pandas 3 was suddenly chosen, nor why the error message is so cryptic, but glad we found a solution in the end!

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

yassine_eal — Tue, 27 Jan 2026 09:13:04 GMT

Hi, I ran into the same error starting late last week.

The issue was caused by dependency version conflicts between my custom package and the libraries preinstalled on the Databricks serverless environment.

I fixed it by pinning all my package dependencies to the versions already provided by the serverless cluster, which resolved the error.

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

dhpaulino_ — Wed, 28 Jan 2026 18:25:44 GMT

Thank you @yassine_eal , it was the same thing here. I haven't analyzed it yet to discover which dependency is breaking the dlt pipeline, but here are my dependencies if it helps anyone.

dependencies = [ "azure-identity>=1.25.1", "httpx>=0.27.0", "msal>=1.34.0", "office365-rest-python-client>=2.6.2", "openpyxl>=3.1.5", "pandas>=2.3.3", "tenacity>=9.0.0", ]

Re: Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

liquibricks — Thu, 29 Jan 2026 10:37:22 GMT

@dhpaulino_ you need to include the upper limit on your pandas library to ensure that v3 is not installed:

"pandas>=2.3.3,<3"