https://community.databricks.com/t5/data-engineering/what-is-best-practice-to-handle-the-concurrency-issue-in-batch/m-p/4140#M940 <P>Normally, our ELT framework takes in batches one by one and loads the data into target tables. But if more than one batches come in at the same time, the framework will break due to the concurrency issue that multiple sources are trying to write the same tables at the same time. </P><P>what is the best way or standard procedure to solve this concurrency issue? Thanks!</P> Fri, 19 May 2023 16:54:28 GMT andrew0117 2023-05-19T16:54:28Z https://community.databricks.com/t5/data-engineering/what-is-best-practice-to-handle-the-concurrency-issue-in-batch/m-p/4140#M940 <P>Normally, our ELT framework takes in batches one by one and loads the data into target tables. But if more than one batches come in at the same time, the framework will break due to the concurrency issue that multiple sources are trying to write the same tables at the same time. </P><P>what is the best way or standard procedure to solve this concurrency issue? Thanks!</P> Fri, 19 May 2023 16:54:28 GMT https://community.databricks.com/t5/data-engineering/what-is-best-practice-to-handle-the-concurrency-issue-in-batch/m-p/4140#M940 andrew0117 2023-05-19T16:54:28Z https://community.databricks.com/t5/data-engineering/what-is-best-practice-to-handle-the-concurrency-issue-in-batch/m-p/4141#M941 <P>you can partition you table to avoid the changes of getting this exception.</P> Wed, 14 Jun 2023 21:25:50 GMT https://community.databricks.com/t5/data-engineering/what-is-best-practice-to-handle-the-concurrency-issue-in-batch/m-p/4141#M941 jose_gonzalez 2023-06-14T21:25:50Z