Databricks Community

Trying to create a Spark Declarative Pipeline with a Materialized View based on a Streaming Table. The Streaming Table works fine, but when trying to add a Materialized View the pipeline fails to initialize with:Your table schema requires manually en...

We have a Lakeflow Spark Declarative Pipeline using the new PySpark Pipelines API. This was working fine until about 7am (Central European) this morning when the pipeline started failing with a PYTHON.NAME_ERROR: name 'kdf' is not defined. Did you me...

I'm trying to define a job for a pipeline using the Asset Bundle Python SDK. I created the pipeline first (using the SDK) and i'm now trying to add the Job. The DAB validates and deploys successfully, but when I run the Job i get an error: UNAUTHORIZ...

We have a pipeline in a job which dynamically creates a set of streaming tables based on a list of kafka topics like this:       # inside a loop      @DP.table(name=table_name, comment=markdown_info)      def topic_flow(topic_name=topic_name):       ...

We are testing an ingestion from kafka to databricks using a streaming table. The streaming table was created by a DAB deployed to "production" which runs as a service principal. This means the service principal is the "owner" of the table.We now wan...