DLT Pipeline Error -key not found: all_info_dlt_cx_utils_cod resulting in a NoSuchElementException.

[ittzzmalind]

Databricks ETL pipeline, specifically an error with the @DP.expectorfail decorator causing the pipeline update to fail. The error message indicated a 'key not found: all_info_dlt_cx_utils_cod ' resulting in a NoSuchElementException.

Note: if we comment @DP.expect_or_fail code works,

below is the sample code:

def get_parquet_stream(parquet_file: str) -> DataFrame:

    """

    Helper function to read parquet files as a streaming source.

    """

    schema_hints = "optimise_date DATE, account_name STRING"

    return utils.get_parquet_stream_with_timestamp_ntz(

        spark=spark,

        parquet_folder=rebal_pack_parquet_folder,

        parquet_file=parquet_file,

        schema_hints=schema_hints

    )

@DP.table(name="all_info_dlt_cx_utils_code")

@DP.expect_or_fail("key_columns_not_null", "optimise_date IS NOT NULL AND account_name IS NOT NULL")

def load_rebal_pack_all_info():

    """

    Creates `all_info` streaming table by reading parquet files.

    """

    return get_parquet_stream("rebal_pack_all_info.parquet")

[MoJaMa]

I don't know if it's a copy paste error on your side but you reference an error: 

'key not found: all_info_dlt_cx_utils_cod '

I notice it's missing an "e".

Can you check the code-base for any typos? Maybe there's a typo in a dlt.read() or spark.table() call referencing this table incorrectly instead of "all_info_dlt_cx_utils_code"?

If you can share more of the code/error we can see if there could be anything else.