topic Re: How to enable spark.shuffle.compress in spark 3.3.0 or above versions? in Data Engineering

How to enable spark.shuffle.compress in spark 3.3.0 or above versions?

Ankith — Thu, 11 May 2023 11:51:55 GMT

when I try to set that I got a following error, appreciate your comments, thanks you in advance.

Re: How to enable spark.shuffle.compress in spark 3.3.0 or above versions?

Anonymous — Sat, 13 May 2023 16:22:44 GMT

@Ankith Patlolla :

The reason you're getting the error "cannot modify the value of a spark config - spark.shuffle.compress" is because you're trying to modify a Spark configuration property that is marked as final and can't be changed after the Spark application has started.

In Databricks, the recommended way to set Spark configuration properties is through the spark.conf.set() method in a notebook or job. However, some Spark configuration properties, like spark.shuffle.compress, cannot be modified after Spark has started. If you need to set spark.shuffle.compress

in Databricks, you can set it in the Spark configuration of the cluster. Here's how you can do it:

Go to your Databricks workspace.
Click on the "Clusters" tab.
Select the cluster for which you want to set the spark.shuffle.compress property.
Click on the "Edit" button.
Scroll down to the "Spark" section and expand it.
Click on the "Advanced Options" tab.
In the "Spark Config" field, add the following line: spark.shuffle.compress true
Click on the "Save" button.

This will set the spark.shuffle.compress property for the selected cluster.

Re: How to enable spark.shuffle.compress in spark 3.3.0 or above versions?

Anonymous — Mon, 22 May 2023 06:57:16 GMT

Hi @Ankith Patlolla

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!