LuisRSanchez
Databricks Partner

Hi

this operation should take seconds because it use the precomputed statistics for the table. Then few elements to verify:

  • if the data type is datetime or integer should work, if it is string data type then it needs to read all data.
  • verify the column position, normally delta lake only create statistics for the some columns (i think first 15), if the column is not at the list of column to precompute stats then it needs to read all data.

if you need to read all data, then i saw in the image that only 4 task are running means 4 cores, then i would recommend to use a bigger cluster  in memory and cores (scale up) with fewer nodes to reduce the shuffle.

View solution in original post