aladda
Databricks Employee
Databricks Employee

You’re likely running into the issue described here and a solution to it as well. While Delta does support concurrent writers to separate partitions of a table, depending on your query structure join/filter/where in particular, there may still be a need to scan the entire table. Solution typically is to have explicit filtering on the partition columns (which you’d also be using in your joins).

View solution in original post