miklos
Databricks Employee
Databricks Employee

The reason this causes a problem is that we're reading and writing to the same path that we're trying to overwrite. This causes an issue since the data cannot be stream into the same directory we're attempting to overwrite.

I'd recommend adding a temporary location to do rewrite this data.

View solution in original post