Apparently the issue was caused by a change in Azure, which was hotfixed cfr https://learn.microsoft.com/en-us/answers/questions/2007592/how-to-fix-the-specified-path-already-ex....

Checkpoints work fine now. We included a recovery attempt in our workflow that restarts streams while resetting checkpoints whenever they fail for whatever reason.