DLT change in hive metastore destination to unity catalog

[Mauro]

A change recently came out in which Databricks necessarily requires using the Unity Catalog as the output of a DLT because previously it was HiveMetaStore. At first I was working using CDC plus expectations which resulted in the "allow_expectations_col" column that allowed me to take actionable action on the applied rules, now this change "omits" this column, therefore, I cannot perform actionable actions, help me understand how to solve this or if there is an alternative to this.

[mark_ott]

Databricks has recently enforced Unity Catalog as the output target for Delta Live Tables (DLT), replacing the legacy Hive Metastore approach. As a result, the familiar "allow_expectations_col" column, which was automatically added to help track and take action on violated expectations, is no longer emitted by default in Unity Catalog-managed tables.​

Why "allow_expectations_col" Is Omitted

Unity Catalog emphasizes streamlined governance, secure data management, and schema standardization across all pipeline outputs. With this migration, the "allow_expectations_col" column that previously allowed you to see and act on rule violations in each output row is not supported or automatically generated in Unity Catalog tables.​

Alternatives and Solutions

• Event Log & Metrics: Instead of relying on row-level markers like "allow_expectations_col," Databricks recommends monitoring expectation violations through the pipeline event logs and the data quality metrics exposed via the Jobs & Pipelines UI or by querying the Lakeflow pipeline event log directly. These logs will show aggregate counts of expectation passes and failures but are not row-level annotations.​

• Retaining Invalid Records: If actionable row-level tracking is required, you can modify how you apply expectations:

  • Use the expect decorator or SQL EXPECT constraint without specifying DROP ROW to retain both valid and invalid records, then query metrics and review violations via the UI or logs.​

  • Use a separate table or output to log failed records, either as part of your DLT workflow (for example, using an “exceptions” table) or via an external data quality tool such as Great Expectations. This allows you to collect and act on failed records outside the main DLT output.​

• Custom Handling Using Great Expectations: For deeper profiling and actionable row-level violations, consider integrating a process that runs Great Expectations validations on your DLT outputs. While this introduces a second technology, it allows for custom tagging and handling failed records, maintaining the actionable quality of prior "allow_expectations_col" behavior.​

• Alternative Table Design: It's possible to create manual audit columns within your transformations and flag records as violating certain expectations, mimicking the old behavior even in Unity Catalog. This would require explicitly adding such columns as part of your DLT pipeline’s transformation logic.

Summary Table: Options for Handling Violations

Alternative	Row-level Visibility	Unity Catalog Supported	Aggregate Metrics	Actionable Records
Pipeline Event Log	No	Yes	Yes	No
Exceptions Table	Yes	Yes (manual)	N/A	Yes
Great Expectations	Yes	Yes (external process)	Yes	Yes
Custom Audit Column	Yes	Yes (manual code)	N/A	Yes

Databricks is unlikely to reintroduce the automatic marker column due to Unity Catalog's design for standardization and compliance, so customizing your pipeline for row-level violation visibility is the suggested path.​