topic Data skipping statistics column datatype constraint in Data Engineering

Data skipping statistics column datatype constraint

SagarJi — Fri, 04 Oct 2024 01:26:25 GMT

Is there any column datatype constraint for the first 32 columns used for the stats that help data skipping?

Re: Data skipping statistics column datatype constraint

SagarJi — Fri, 04 Oct 2024 02:03:30 GMT

I am using Apache Spark 3.5.1 and deltalake version 3.1.0

Re: Data skipping statistics column datatype constraint

Mounika_Tarigop — Sat, 07 Dec 2024 06:24:14 GMT

There are no specific column datatype constraints for the first 32 columns used for the statistics that help with data skipping in Databricks. However, data skipping is not supported for partition columns.