cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to enable spark.shuffle.compress in spark 3.3.0 or above versions?

Ankith
New Contributor

when I try to set that I got a following error, appreciate your comments, thanks you in advance.

image.png

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

@Ankith Patlolla​ :

The reason you're getting the error "cannot modify the value of a spark config - spark.shuffle.compress" is because you're trying to modify a Spark configuration property that is marked as final and can't be changed after the Spark application has started.

In Databricks, the recommended way to set Spark configuration properties is through the spark.conf.set() method in a notebook or job. However, some Spark configuration properties, like spark.shuffle.compress, cannot be modified after Spark has started. If you need to set spark.shuffle.compress

in Databricks, you can set it in the Spark configuration of the cluster. Here's how you can do it:

  1. Go to your Databricks workspace.
  2. Click on the "Clusters" tab.
  3. Select the cluster for which you want to set the spark.shuffle.compress property.
  4. Click on the "Edit" button.
  5. Scroll down to the "Spark" section and expand it.
  6. Click on the "Advanced Options" tab.
  7. In the "Spark Config" field, add the following line: spark.shuffle.compress true
  8. Click on the "Save" button.

This will set the spark.shuffle.compress property for the selected cluster.

View solution in original post

2 REPLIES 2

Anonymous
Not applicable

@Ankith Patlolla​ :

The reason you're getting the error "cannot modify the value of a spark config - spark.shuffle.compress" is because you're trying to modify a Spark configuration property that is marked as final and can't be changed after the Spark application has started.

In Databricks, the recommended way to set Spark configuration properties is through the spark.conf.set() method in a notebook or job. However, some Spark configuration properties, like spark.shuffle.compress, cannot be modified after Spark has started. If you need to set spark.shuffle.compress

in Databricks, you can set it in the Spark configuration of the cluster. Here's how you can do it:

  1. Go to your Databricks workspace.
  2. Click on the "Clusters" tab.
  3. Select the cluster for which you want to set the spark.shuffle.compress property.
  4. Click on the "Edit" button.
  5. Scroll down to the "Spark" section and expand it.
  6. Click on the "Advanced Options" tab.
  7. In the "Spark Config" field, add the following line: spark.shuffle.compress true
  8. Click on the "Save" button.

This will set the spark.shuffle.compress property for the selected cluster.

Anonymous
Not applicable

Hi @Ankith Patlolla​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!