- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2021 04:23 PM
I would like to know how do disable broadcast join in my job to avoid this error message. Is there a Spark configuration?
- Labels:
-
Error Message
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2021 04:25 PM
You can disable broadcast join by adding the following Spark configuration to you notebook:
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
In addition, you can also add this configuration to your cluster:
spark.sql.autoBroadcastJoinThreshold -1
You can disable it at notebook level or cluster level.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2021 04:25 PM
You can disable broadcast join by adding the following Spark configuration to you notebook:
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
In addition, you can also add this configuration to your cluster:
spark.sql.autoBroadcastJoinThreshold -1
You can disable it at notebook level or cluster level.