What is the maximum limit of data that can be broadcasted using broadcast join

[brickster_2018 · Accepted solution]

By default, only 10 MB of data can be broadcasted.

spark.sql.autoBroadcastJoinThreshold can be increased up to 8GB

There is an upper limit in terms of records as well. We can't broadcast more than 512m records. So its either 512m records or 8GB which ever limit hits first

[lchari]

Is the limit per "table/dataframe" or for all tables/dataframes put together?

The driver collects the data from all executors (which are having the respective table or dataframe) and distributes to all executors. When will the memory be released in both driver and executor? Or does it hold on to this memory through out the pipeline/application?