topic Re: How do I reduce the size of a hive table's S3 bucket in Data Engineering

How do I reduce the size of a hive table's S3 bucket

dotan — Mon, 06 Mar 2023 16:27:58 GMT

I have a hive table in Delta format with over 1B rows, when I check the Data Explorer in the SQL section of Databricks it notes that the table size is 139.3GiB with 401 files but when I check the S3 bucket where the files are located (dbfs:/user/hive/warehouse/large_table) it's over 110TB and contains over 100K files.

Is it possible to reduce the size of the S3 bucket without losing any data in the table?

Re: How do I reduce the size of a hive table's S3 bucket

apingle — Mon, 06 Mar 2023 21:52:20 GMT

When you run updates, deletes etc on a delta table, new files are created. However, the old files are not automatically deleted. This is to allow for features like time travel on the Delta tables.

In order to delete older files for a delta table, you can use the vacuum command.

https://docs.databricks.com/sql/language-manual/delta-vacuum.html

Re: How do I reduce the size of a hive table's S3 bucket

dotan — Tue, 07 Mar 2023 19:47:07 GMT

That's great, thanks. It reduced the size of the bucket from 110TB to 7TB