topic Re: Metric views materialization failure in Data Engineering

Metric views materialization failure

alejandro_jaram — Thu, 10 Sep 2026 16:44:12 GMT

Hi, I have metric views with materilization option, it was running fine but today it just failed without any change to my metric view definition, all metric views are failing so I guess is something general. Based on limited logs (users can´t fully audit DLT code) looks like clustering columns required by user are not being added to stats columns.

Thank you in advance

Here is an example of how materializations are defined in yml file:

materialization:
  schedule: CRON '30 0 5 ? * Mon' AT TIME ZONE 'UTC'
  mode: relaxed
  materialized_views:
    - name: baseline
      type: unaggregated
      cluster_by:
        auto: false
        cols:
          - Travel Month
          - Operating Carrier
          - Departure Airport IATA
          - Arrival Airport IATA

Re: Metric views materialization failure

osingh — Fri, 11 Sep 2026 11:12:41 GMT

This error happens because Delta's Liquid Clustering requires column statistics to be collected for every column specified in your cluster_by.cols list (Travel Month, Operating Carrier, Departure Airport IATA, Arrival Airport IATA), but Enzyme (the internal engine handling Metric Views and Materialized Views) omitted Operating Carrier from the stats schema during its internal initialization query (__enzyme_row_id__). Since this started failing across all metric views without any definition changes, it points to a recent Databricks backend runtime update tightening stats validation for DLT pipelines.

Here are few steps you can try:

1. Force stats collection on the clustering columns
Explicitly define the exact columns Delta must collect stats for using table properties on your target metric view:

ALTER TABLE `intelligence_dv`.`gold`.`bookings_adjusted_segments_metric_view` SET TBLPROPERTIES ( 'delta.dataSkippingStatsColumns' = 'Travel Month, Operating Carrier, Departure Airport IATA, Arrival Airport IATA' );

Verification: Run the pipeline again to verify that the initialization phase passes without the missing stats error.

2. Perform a Full Refresh on the DLT pipeline
Backend runtime updates can cause internal Enzyme projections and Delta metadata to get out of sync. In your pipeline UI, click Select tables for refresh and run a Full Refresh.
Verification: Check the pipeline event log to ensure all tables rebuild cleanly from scratch.

3. Increase the indexed columns threshold
Delta Lake stops collecting statistics after the first 32 columns by default. If Operating Carrier sits further down in your schema, bump the limit:

ALTER TABLE `intelligence_dv`.`gold`.`bookings_adjusted_segments_metric_view` SET TBLPROPERTIES ('delta.dataSkippingNumIndexedCols' = '64');

Verification: Trigger the pipeline and check the driver logs to confirm stats are generated for all clustering columns.

4. Workaround: Switch to Automatic Clustering
If explicit column clustering continues to fail due to Enzyme dropping stats context, update your YAML definition to let Databricks manage clustering automatically:

cluster_by: auto: true

Verification: Re-deploy the YAML definition and confirm the scheduled materialization job runs to completion.

Hope this should work!

PS-If none of these work, I'd recommend raising a ticket with Databricks support, as this is likely a bug introduced in a recent Enzyme engine release for Metric Views.

Re: Metric views materialization failure

Louis_Frolio — Fri, 09 Oct 2026 13:51:56 GMT

Hello @alejandro_jaram, I took a look at both internal and external documentation and here is what I found.

First, credit to @osingh for pointing at clustering and statistics validation. Liquid clustering does require file stats on every clustering key, and the error in your screenshot looks like the DELTA_CLUSTERING_COLUMN_MISSING_STATS family. That said, the post doesn't prove a backend regression on its own, and I wouldn't lean on the internal engine details in that reply as a diagnosis. Every metric view failing on the same day with no YAML changes is decent evidence of a platform-side change, but the exact error code, update ID, and runtime are what confirm it.

One correction that'll save you some frustration. A metric view is a view, not a Delta table, so ALTER TABLE ... SET TBLPROPERTIES against the metric view name will fail. The materialized views underneath it are created and maintained by a managed Lakeflow pipeline, and you don't own their table properties either. delta.dataSkippingStatsColumns and ANALYZE TABLE ... COMPUTE DELTA STATISTICS are the right tools for tables you create yourself, just not here.

What you can do:

Get the real error. Run DESCRIBE EXTENDED catalog.schema.metric_view and follow the pipeline link in the Refresh Information section, or use the link under Refresh Schedule on the metric view's Overview tab in Catalog Explorer. The pipeline event log gives you the full error code and update ID.
Validate the YAML. The names in cluster_by.cols must be metric view field names, not display labels or source column names. Also confirm your unaggregated materialization doesn't carry dimensions or measures, since that type materializes the whole model.
Try a full refresh: REFRESH MATERIALIZED VIEW catalog.schema.metric_view FULL;. If a runtime update left the materialization metadata out of step with the definition, this is the cheapest way to find out.
Isolate it. Remove cluster_by from one materialization and refresh. If that succeeds, add the columns back one at a time, or switch to cluster_by: auto: true as osingh suggested. Either option changes physical layout and performance, so treat it as a diagnostic or a stopgap rather than a fix.
If it still fails after a clean YAML and a full refresh, open a support ticket with the pipeline ID, event log entries, runtime, YAML, and the fact that all your metric views broke at once. Since the pipeline is managed, support can see what you can't.

If you learn the root cause, please post it back here.

References:

Materialization for metric views: https://docs.databricks.com/aws/en/uc-semantics/metric-views/materialization
Metric view YAML reference: https://docs.databricks.com/aws/en/uc-semantics/metric-views/yaml-reference
REFRESH MATERIALIZED VIEW: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-refresh-full
Liquid clustering: https://docs.databricks.com/aws/en/tables/clustering
Data skipping and statistics columns: https://docs.databricks.com/aws/en/tables/data-skipping
KB: DELTA_CLUSTERING_COLUMN_MISSING_STATS: https://kb.databricks.com/delta/delta_clustering_column_missing_stats-error-when-attempting-to-define-liquid-clustering-for-a-delta-table

Regards, Louis.