cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Aviral-Bhardwaj
by • Esteemed Contributor III
  • 2017 Views
  • 1 replies
  • 19 kudos

�� Deltalake Vs Datalake in Databricks ��Delta Lake Databricks Delta Lake is an open-source storage layer that sits on top of existing d...

Deltalake Vs Datalake in Databricks Delta Lake DatabricksDelta Lake is an open-source storage layer that sits on top of existing data lake storage, such as Azure Data Lake Store or Amazon S3. It provides a more robust and scalable alternative to tra...

Data Engineering
Reply
  • 2017 Views
  • 1 replies
  • 19 kudos
View Replies
gbrueckl
by • Contributor II
  • 9230 Views
  • 8 replies
  • 9 kudos

Slow performance of VACUUM on Azure Data Lake Store Gen2

We need to run VACCUM on one of our biggest tables to free the storage. According to our analysis using VACUUM bigtable DRY RUN this affects 30M+ files that need to be deleted.If we run the final VACUUM, the file-listing takes up to 2h (which is OK) ...

Data Engineering
Reply
  • 9230 Views
  • 8 replies
  • 9 kudos
View Replies
Latest Reply
Deepak_Bhutada
Databricks Employee
  • 9 kudos

@Gerhard Brueckl​ we have seen near 80k-120k file deletions in Azure per hour while doing a VACUUM on delta tables, it's just that the vacuum is slower in azure and S3. It might take some time as you said when deleting the files from the delta path. ...

  • 9 kudos
7 More Replies
Labels