DLT Pipeline Error -key not found: all_info_dlt_cx_utils_cod resulting in a NoSuchElementException.

ittzzmalind
New Contributor III

‎03-27-2026 10:42 AM

Databricks ETL pipeline, specifically an error with the @DP.expectorfail decorator causing the pipeline update to fail. The error message indicated a 'key not found: all_info_dlt_cx_utils_cod ' resulting in a NoSuchElementException.

Note: if we comment @DP.expect_or_fail code works,

below is the sample code:

def get_parquet_stream(parquet_file: str) -> DataFrame:
    """
    Helper function to read parquet files as a streaming source.
    """
    schema_hints = "optimise_date DATE, account_name STRING"
    return utils.get_parquet_stream_with_timestamp_ntz(
        spark=spark,
        parquet_folder=rebal_pack_parquet_folder,
        parquet_file=parquet_file,
        schema_hints=schema_hints
    )

@DP.table(name="all_info_dlt_cx_utils_code")
@DP.expect_or_fail("key_columns_not_null", "optimise_date IS NOT NULL AND account_name IS NOT NULL")
def load_rebal_pack_all_info():
    """
    Creates `all_info` streaming table by reading parquet files.
    """
    return get_parquet_stream("rebal_pack_all_info.parquet")
Labels:
0 Kudos

MoJaMa
Databricks Employee
Databricks Employee

‎03-27-2026 02:18 PM

I don't know if it's a copy paste error on your side but you reference an error: 

'key not found: all_info_dlt_cx_utils_cod '

I notice it's missing an "e".

Can you check the code-base for any typos? Maybe there's a typo in a dlt.read() or spark.table() call referencing this table incorrectly instead of "all_info_dlt_cx_utils_code"?

If you can share more of the code/error we can see if there could be anything else.

0 Kudos

ittzzmalind
New Contributor III

‎04-02-2026 07:27 AM

@MoJaMa Thanks for the reply, The issue was in the code, corrected code worked

0 Kudos