Khaja_Zaffer
Esteemed Contributor

Understood. @VaderKB 

Please check this documentation around this : https://docs.databricks.com/aws/en/delta/best-practices
Databricks recommends frequently running the OPTIMIZE command to compact small files.

Please try to run this: 

-- For a non-partitioned table
OPTIMIZE my_delta_table;

-- For a table partitioned by 'date', optimizing the last 2 days
OPTIMIZE my_delta_table WHERE date >= current_date() - INTERVAL 2 DAY;

I am waiting other in the forum to confirm this solution. @VaderKB  I am sure this is not production environment right?