After checking with Microsoft support: "this issue is happening only temporarily due to the browser cache and the issue is resolved after a few mins and we don't see that after the browser cache has been updated. This could be expected with the eventual consistency."