02-28-2022 05:43 AM
Hi! I'm starting to test configs on DataBricks, for example, to avoid corrupting data if two processes try to write at the same time:
.config('spark.databricks.delta.multiClusterWrites.enabled', 'false')
Or if I need more partitions than default
.config('spark.databricks.adaptive.autoOptimizeShuffle.enabled', 'true')
Is there another recommended default setting? (then goes the tunning for each job)
Thanks!
02-28-2022 09:39 AM
Delta tables do have optimistic concurrency control. So if two processes are trying to write to the same table it does its best to handle both but if the transactions are conflicts then it will fail. You can also change the isolation levels if you want to enforce more control on that.
02-28-2022 09:39 AM
Delta tables do have optimistic concurrency control. So if two processes are trying to write to the same table it does its best to handle both but if the transactions are conflicts then it will fail. You can also change the isolation levels if you want to enforce more control on that.
03-01-2022 12:29 AM
Exactly. You can easy verify that as commits are written to separate files in delta log.
Regarding:
.config('spark.databricks.adaptive.autoOptimizeShuffle.enabled', 'true')
and other spark optimization solutions please watch databricks video about that https://www.youtube.com/watch?v=daXEp4HmS-E
03-17-2022 02:35 AM
Hi @Alejandro Martinez , How is it going? Did the docs help you anyhow?
03-17-2022 06:07 AM
It helped but still testing different configurations, thank you!
04-28-2022 09:27 AM
Hey there @Alejandro Martinez
Hope everything is going well.
Just wanted to see if you were able to find an answer to your question. If yes, would you be happy to let us know and mark it as best so that other members can find the solution more quickly?
Cheers!
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.