topic Re: DLT Pipeline Error -key not found: all_info_dlt_cx_utils_cod resulting in a NoSuchElementExcepti in Data Engineering

DLT Pipeline Error -key not found: all_info_dlt_cx_utils_cod resulting in a NoSuchElementException.

ittzzmalind — Fri, 27 Mar 2026 17:42:04 GMT

Databricks ETL pipeline, specifically an error with the @DP.expectorfail decorator causing the pipeline update to fail. The error message indicated a 'key not found: all_info_dlt_cx_utils_cod ' resulting in a NoSuchElementException.

Note: if we comment @DP.expect_or_fail code works,

below is the sample code:

def get_parquet_stream(parquet_file: str) -> DataFrame:

"""

Helper function to read parquet files as a streaming source.

"""

schema_hints = "optimise_date DATE, account_name STRING"

return utils.get_parquet_stream_with_timestamp_ntz(

spark=spark,

parquet_folder=rebal_pack_parquet_folder,

parquet_file=parquet_file,

schema_hints=schema_hints

)

@DP.table(name="all_info_dlt_cx_utils_code")

@DP.expect_or_fail("key_columns_not_null", "optimise_date IS NOT NULL AND account_name IS NOT NULL")

def load_rebal_pack_all_info():

"""

Creates `all_info` streaming table by reading parquet files.

"""

return get_parquet_stream("rebal_pack_all_info.parquet")

Re: DLT Pipeline Error -key not found: all_info_dlt_cx_utils_cod resulting in a NoSuchElementExcepti

MoJaMa — Fri, 27 Mar 2026 21:18:42 GMT

I don't know if it's a copy paste error on your side but you reference an error:

'key not found: all_info_dlt_cx_utils_cod '

I notice it's missing an "e".

Can you check the code-base for any typos? Maybe there's a typo in a dlt.read() or spark.table() call referencing this table incorrectly instead of "all_info_dlt_cx_utils_code"?

If you can share more of the code/error we can see if there could be anything else.

Re: DLT Pipeline Error -key not found: all_info_dlt_cx_utils_cod resulting in a NoSuchElementExcepti

ittzzmalind — Thu, 02 Apr 2026 14:27:53 GMT

@MoJaMa Thanks for the reply, The issue was in the code, corrected code worked