topic Re: Are there any recommended spark config settings for Delta/Databricks? in Data Engineering

Are there any recommended spark config settings for Delta/Databricks?

alejandrofm — Mon, 28 Feb 2022 13:43:25 GMT

Hi! I'm starting to test configs on DataBricks, for example, to avoid corrupting data if two processes try to write at the same time:

.config('spark.databricks.delta.multiClusterWrites.enabled', 'false')

Or if I need more partitions than default

.config('spark.databricks.adaptive.autoOptimizeShuffle.enabled', 'true')

Is there another recommended default setting? (then goes the tunning for each job)

Thanks!

Re: Are there any recommended spark config settings for Delta/Databricks?

Ryan_Chynoweth — Mon, 28 Feb 2022 17:39:31 GMT

Delta tables do have optimistic concurrency control. So if two processes are trying to write to the same table it does its best to handle both but if the transactions are conflicts then it will fail. You can also change the isolation levels if you want to enforce more control on that.

Re: Are there any recommended spark config settings for Delta/Databricks?

Hubert-Dudek — Tue, 01 Mar 2022 08:29:03 GMT

Exactly. You can easy verify that as commits are written to separate files in delta log.

Regarding:

.config('spark.databricks.adaptive.autoOptimizeShuffle.enabled', 'true')

and other spark optimization solutions please watch databricks video about that https://www.youtube.com/watch?v=daXEp4HmS-E

Re: Are there any recommended spark config settings for Delta/Databricks?

alejandrofm — Thu, 17 Mar 2022 13:07:31 GMT

It helped but still testing different configurations, thank you!

Re: Are there any recommended spark config settings for Delta/Databricks?

Anonymous — Thu, 28 Apr 2022 16:27:40 GMT

Hey there @Alejandro Martinez

Hope everything is going well.

Just wanted to see if you were able to find an answer to your question. If yes, would you be happy to let us know and mark it as best so that other members can find the solution more quickly?

Cheers!