05-11-2023 04:51 AM
05-13-2023 09:22 AM
@Ankith Patlolla :
The reason you're getting the error "cannot modify the value of a spark config - spark.shuffle.compress" is because you're trying to modify a Spark configuration property that is marked as final and can't be changed after the Spark application has started.
In Databricks, the recommended way to set Spark configuration properties is through the spark.conf.set() method in a notebook or job. However, some Spark configuration properties, like spark.shuffle.compress, cannot be modified after Spark has started. If you need to set spark.shuffle.compress
in Databricks, you can set it in the Spark configuration of the cluster. Here's how you can do it:
This will set the spark.shuffle.compress property for the selected cluster.
05-13-2023 09:22 AM
@Ankith Patlolla :
The reason you're getting the error "cannot modify the value of a spark config - spark.shuffle.compress" is because you're trying to modify a Spark configuration property that is marked as final and can't be changed after the Spark application has started.
In Databricks, the recommended way to set Spark configuration properties is through the spark.conf.set() method in a notebook or job. However, some Spark configuration properties, like spark.shuffle.compress, cannot be modified after Spark has started. If you need to set spark.shuffle.compress
in Databricks, you can set it in the Spark configuration of the cluster. Here's how you can do it:
This will set the spark.shuffle.compress property for the selected cluster.
05-21-2023 11:57 PM
Hi @Ankith Patlolla
Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?
This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.