topic I read that Delta supports concurrent writes to separate partitions of the table but I'm getting an error when doing so in Data Engineering

I read that Delta supports concurrent writes to separate partitions of the table but I'm getting an error when doing so

aladda — Mon, 21 Jun 2021 20:09:36 GMT

I’m running 3 separate dbt processes in parallel. all of them are reading data from different databrick databases, creating different staging tables by using dbt alias, but they all at the end update/insert to the same target table. the 3 processes run well except in the last step. sometimes one of the process fails in the last step and sometimes 2 of them fail.

Re: I read that Delta supports concurrent writes to separate partitions of the table but I'm getting an error when doing so

aladda — Mon, 21 Jun 2021 20:10:01 GMT

You’re likely running into the issue described here and a solution to it as well. While Delta does support concurrent writers to separate partitions of a table, depending on your query structure join/filter/where in particular, there may still be a need to scan the entire table. Solution typically is to have explicit filtering on the partition columns (which you’d also be using in your joins).