cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Live Tables maintenance schedule

vroste
New Contributor III

I have a DLT that runs every day and an automatically executed maintenance job that runs within 24 hours every day. The maintenance operations are costly, is it possible to change the schedule to once a week or so?

1 ACCEPTED SOLUTION

Accepted Solutions

Kaniz
Community Manager
Community Manager

Hi @vrosteBased on the information provided, it is impossible to directly change the frequency of the automatic maintenance tasks performed by Delta Live Tables (DLT) from every 24 hours to once a week. The system is designed to perform maintenance tasks within 24 hours of a table being updated to improve query performance and reduce costs by removing old versions of tables.

However, you might consider a few indirect approaches:

โ€ข You could control the frequency of table updates, as maintenance tasks are performed only if a pipeline update has run 24 hours before the maintenance tasks are scheduled. The maintenance tasks will also run less frequently if the table is updated less frequently.

โ€ข You can turn off the OPTIMIZE operation for a table by setting pipelines.autoOptimize.managed = false in the table properties for the table. This might reduce the cost of maintenance operations, although it could also affect query performance. Remember to apply the necessary security configurations to both the default and maintenance clusters to ensure the maintenance cluster has the required storage location access.

Sources:
- [Docs: index](https://docs.databricks.com/delta-live-tables/index.html

View solution in original post

1 REPLY 1

Kaniz
Community Manager
Community Manager

Hi @vrosteBased on the information provided, it is impossible to directly change the frequency of the automatic maintenance tasks performed by Delta Live Tables (DLT) from every 24 hours to once a week. The system is designed to perform maintenance tasks within 24 hours of a table being updated to improve query performance and reduce costs by removing old versions of tables.

However, you might consider a few indirect approaches:

โ€ข You could control the frequency of table updates, as maintenance tasks are performed only if a pipeline update has run 24 hours before the maintenance tasks are scheduled. The maintenance tasks will also run less frequently if the table is updated less frequently.

โ€ข You can turn off the OPTIMIZE operation for a table by setting pipelines.autoOptimize.managed = false in the table properties for the table. This might reduce the cost of maintenance operations, although it could also affect query performance. Remember to apply the necessary security configurations to both the default and maintenance clusters to ensure the maintenance cluster has the required storage location access.

Sources:
- [Docs: index](https://docs.databricks.com/delta-live-tables/index.html