Are there any recommended spark config settings for Delta/Databricks?

alejandrofm
Valued Contributor

‎02-28-2022 05:43 AM

Hi! I'm starting to test configs on DataBricks, for example, to avoid corrupting data if two processes try to write at the same time:

.config('spark.databricks.delta.multiClusterWrites.enabled', 'false')

Or if I need more partitions than default

.config('spark.databricks.adaptive.autoOptimizeShuffle.enabled', 'true')

Is there another recommended default setting? (then goes the tunning for each job)

Thanks!

Labels:
0 Kudos

Ryan_Chynoweth
Databricks Employee
Databricks Employee

‎02-28-2022 09:39 AM

Delta tables do have optimistic concurrency control. So if two processes are trying to write to the same table it does its best to handle both but if the transactions are conflicts then it will fail. You can also change the isolation levels if you want to enforce more control on that.

View solution in original post

Hubert-Dudek
Databricks MVP
In response to Ryan_Chynoweth

‎03-01-2022 12:29 AM

Exactly. You can easy verify that as commits are written to separate files in delta log.

Regarding:

.config('spark.databricks.adaptive.autoOptimizeShuffle.enabled', 'true')

and other spark optimization solutions please watch databricks video about that https://www.youtube.com/watch?v=daXEp4HmS-E


My blog: https://databrickster.medium.com/

alejandrofm
Valued Contributor
In response to Retired_mod

‎03-17-2022 06:07 AM

It helped but still testing different configurations, thank you!

0 Kudos

Anonymous
Not applicable

‎04-28-2022 09:27 AM

Hey there @Alejandro Martinez​ 

Hope everything is going well.

Just wanted to see if you were able to find an answer to your question. If yes, would you be happy to let us know and mark it as best so that other members can find the solution more quickly?

Cheers!

0 Kudos