We have a Lakeflow Spark Declarative Pipeline using the new PySpark Pipelines API. This was working fine until about 7am (Central European) this morning when the pipeline started failing with a PYTHON.NAME_ERROR: name 'kdf' is not defined. Did you mean: 'sdf'?
The code has not changed and nothing in our infrastructure has changed, so i'm not sure why this has suddenly started happening. I've raised an error with Azure (we're on Azure Databricks), but they are slow so i'm shouting into the void here. Any suggestions? 😅
Usage:
from pyspark import pipelines as dp
...
@dp.table(name=f"{mycatalog}.{myschema}.{mytable}")
def create_streaming_table(😞
return (
....
)
