- 6293 Views
- 3 replies
- 3 kudos
Resolved! Slow performance loading checkpoint file?
Using OSS Delta, hopefully this is the right forum for this question:Hey all, I could use some help as I feel like I’m doing something wrong here.I’m streaming from Kafka -> Delta on EMR/S3FS, and am seeing ever-increasingly slow batches.When looking...
- 6293 Views
- 3 replies
- 3 kudos
- 3 kudos
Found the answer through the Slack user group, courtesy of an Adam Binford.I had set `delta.logRetentionDuration='24 HOURS'` but did not set `delta.deletedFileRetentionDuration`, and so the checkpoint file still had all the accumulated tombstones sin...
- 3 kudos
- 5064 Views
- 3 replies
- 3 kudos
Resolved! Roll back to previous version of an AutoLoader checkpoint file
I know to "reset" AutoLoader, you can delete the checkpoint file entirely. I was wondering if it's possible to and how would someone :Get the checkpoint file to a previous version so I can reload certain files that were already processedDelete certai...
- 5064 Views
- 3 replies
- 3 kudos
