Hi,
we have implemented a job that runs on a trigger of a table update. The job worked perfectly, until the source table now have accumulated too many log files and the job isn't running anymore. Only the error message below:
Storage location /abcd/_delta_log/ contains more than the allowed limit of 9999 objects. Remove objects or choose a different location for your table trigger.
We have implemented a retention policy (delta.logRetentionDuration=interval 7 days) for the source table to hopefully correct the issue but it seems it does nothing. Checking we still see log files from several months.
Moreover, each update of the source table seems to produce 8 log files, and the table currently is update each hour. This gives nearly 200 new log files each day, and if we want to update the source table more often we will quickly run into the 9999 file limit.
Are there any suggestions how handle the log file retention? If the logRetentionduration doesn't work, seemingly we can't rely on the TableTrigger function. Can we manually go and delete the files, or implement some s3 retention policy on our own or will this break some logic?
Thanks for any suggestions.
