Optimize Command not performing the bin packing

brickster_2018
Databricks Employee
Databricks Employee

I have a daily OPTIMIZE job running, however, the number of files in the storage is not going down. Looks like the optimize is not helping to reduce the files.

brickster_2018
Databricks Employee
Databricks Employee

The files are not physically removed from the Storage by the optimize command. A VACUUM command has to be executed to achieve the same