I recently ran into this MetadataChangedException. Watching the video @Hubert Dudek​ posted it's pretty clear what is going on: object storage folks not thinking like someone who builds relational database engines built it. That's to be expected. Databricks is wonderful in many ways, but in SQL and relational database engine features like sequences, they're evolving slowly.

I switched to a serial write to get around the problem because of a deadline but we really should open a ticket on this with Databricks to get some clarity on an issue (parallel sequence updates) relational databases solved 50+ years ago. Like the video says, it's a bad idea to store the identity information in schema. Needs to be a something like a separate file with a thread safe approach to updates.