the error disappeared now after a few months.

It could be

1. happens when there is a lot of schema change on db during the initial project stage.
2. there may be some temp security issues
3. may be databricks fixed in the back end
4. appears less issue when schema rarely changes after the project get stable and schema is set