Databricks pipeline fails expectation on execute python script, throws error: Update FAILES

RobFer1985 — Wed, 10 Dec 2025 18:46:09 GMT

Hi Community,

I'm new to Databricks and am trying to make and implement pipeline expectations, The pipelines work without errors and my job works. I've tried multiple ways to implement expectations, sql and python. I keep resolving the errors but end up with the same error. I'm working with the free trial version of Databricks. Is there a limitation to building expectations on the trial version? Are there table permissions in databricks I'm not taking into account? The order_2 table is a streaming table, are there limitations to applying expectations to streaming tables? My python code:

%python from pyspark import pipelines as dp from pyspark.sql.functions import col @dp.table( name="xyntrel_bronze.bronze.orders_2", comment="Orders table with data quality constraints" ) @dp.expect_or_fail("row count > 100", "COUNT(*) > 100") @dp.expect_or_fail("customer_id not null", "customer_id IS NOT NULL") def bronze_table(): return ( spark.readStream.table("xyntrel_bronze.bronze.orders_2") .filter(col("order_id").isNotNull()) )

The complete error in json:

"timestamp": "2025-12-10T18:23:25.679Z", "message": "Update 19907c is FAILED.", "level": "ERROR", "error": { "exceptions": [ { "message": "", "error_class": "_UNCLASSIFIED_PYTHON_COMMAND_ERROR", "short_message": "" } ], "fatal": true }, "details": { "update_progress": { "state": "FAILED" } }, "event_type": "update_progress", "maturity_level": "STABLE" }

Thanks guys!

Re: Databricks pipeline fails expectation on execute python script, throws error: Update FAILES

emma_s — Thu, 11 Dec 2025 13:59:36 GMT

Hey, I think it may be the row_count condition causing the issue. The expectation runs on each row and sees if the record meets the criteria in the expectation, so you're effectively asking count * on each record, which will always evaluate to 1 and therefore always fail your condition. I hope this helps.

topic Databricks pipeline fails expectation on execute python script, throws error: Update FAILES in Data Engineering

Databricks pipeline fails expectation on execute python script, throws error: Update FAILES

Re: Databricks pipeline fails expectation on execute python script, throws error: Update FAILES