Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-09-2024 07:04 AM
I have a process expected to run every two weeks. Throughout the process (~30 notebooks), when I write to a table for the last time in the overall process, I run my vacuum such as below - I'm never running a vac against the same table twice. I've noticed this cleanup process can add substantial time to my notebook run times. My questions;
1. Should I put all cleanups in a separate notebook and run it in a separate workflow.
2. Am I doing it the right way?
#databricksCatalog contains catalog/schema for each modification if it changes.
myTable = databricksCatalog + "my_table_name"
spark.sql(f"VACUUM {myTable} RETAIN 504 HOURS")
spark.sql(f"OPTIMIZE {myTable}")
spark.sql(f"ANALYZE TABLE {myTable} COMPUTE DELTA STATISTICS")Do I need to vac/opt/analyse each time?
Labels:
- Labels:
-
Workflows