Getting broadcast join errors

jose_gonzalez — Fri, 18 Jun 2021 23:23:34 GMT

I would like to know how do disable broadcast join in my job to avoid this error message. Is there a Spark configuration?

Re: Getting broadcast join errors

jose_gonzalez — Fri, 18 Jun 2021 23:25:37 GMT

You can disable broadcast join by adding the following Spark configuration to you notebook:

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)

In addition, you can also add this configuration to your cluster:

spark.sql.autoBroadcastJoinThreshold -1

You can disable it at notebook level or cluster level.