DLT Pipeline Error -key not found: all_info_dlt_cx_utils_cod resulting in a NoSuchElementException.

ittzzmalind
New Contributor III

‎03-27-2026 10:42 AM

Databricks ETL pipeline, specifically an error with the @DP.expectorfail decorator causing the pipeline update to fail. The error message indicated a 'key not found: all_info_dlt_cx_utils_cod ' resulting in a NoSuchElementException.

Note: if we comment @DP.expect_or_fail code works,

below is the sample code:

def get_parquet_stream(parquet_file: str) -> DataFrame:
    """
    Helper function to read parquet files as a streaming source.
    """
    schema_hints = "optimise_date DATE, account_name STRING"
    return utils.get_parquet_stream_with_timestamp_ntz(
        spark=spark,
        parquet_folder=rebal_pack_parquet_folder,
        parquet_file=parquet_file,
        schema_hints=schema_hints
    )

@DP.table(name="all_info_dlt_cx_utils_code")
@DP.expect_or_fail("key_columns_not_null", "optimise_date IS NOT NULL AND account_name IS NOT NULL")
def load_rebal_pack_all_info():
    """
    Creates `all_info` streaming table by reading parquet files.
    """
    return get_parquet_stream("rebal_pack_all_info.parquet")
Labels:
0 Kudos