Getting broadcast join errors

[jose_gonzalez]

I would like to know how do disable broadcast join in my job to avoid this error message. Is there a Spark configuration?

[jose_gonzalez · Accepted solution]

You can disable broadcast join by adding the following Spark configuration to you notebook:

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)

In addition, you can also add this configuration to your cluster:

spark.sql.autoBroadcastJoinThreshold -1

You can disable it at notebook level or cluster level.