https://community.databricks.com/t5/data-engineering/what-is-the-maximum-limit-of-data-that-can-be-broadcasted-using/m-p/18600#M12373 Fri, 25 Jun 2021 21:51:19 GMT brickster_2018 2021-06-25T21:51:19Z https://community.databricks.com/t5/data-engineering/what-is-the-maximum-limit-of-data-that-can-be-broadcasted-using/m-p/18600#M12373 Fri, 25 Jun 2021 21:51:19 GMT https://community.databricks.com/t5/data-engineering/what-is-the-maximum-limit-of-data-that-can-be-broadcasted-using/m-p/18600#M12373 brickster_2018 2021-06-25T21:51:19Z https://community.databricks.com/t5/data-engineering/what-is-the-maximum-limit-of-data-that-can-be-broadcasted-using/m-p/18601#M12374 <P>By default, only 10 MB of data can be broadcasted. </P><P>spark.sql.autoBroadcastJoinThreshold can be increased up to 8GB</P><P>There is an upper limit in terms of records as well. We can't broadcast more than 512m records. So its either 512m records or 8GB which ever limit hits first</P><P></P> Fri, 25 Jun 2021 21:53:47 GMT https://community.databricks.com/t5/data-engineering/what-is-the-maximum-limit-of-data-that-can-be-broadcasted-using/m-p/18601#M12374 brickster_2018 2021-06-25T21:53:47Z https://community.databricks.com/t5/data-engineering/what-is-the-maximum-limit-of-data-that-can-be-broadcasted-using/m-p/99022#M39890 <P>Is the limit per "table/dataframe" or for all tables/dataframes put together?</P><P>The driver collects the data from all executors (which are having the respective table or dataframe) and distributes to all executors. When will the memory be released in both driver and executor? Or does it hold on to this memory through out the pipeline/application?</P> Sat, 16 Nov 2024 14:01:19 GMT https://community.databricks.com/t5/data-engineering/what-is-the-maximum-limit-of-data-that-can-be-broadcasted-using/m-p/99022#M39890 lchari 2024-11-16T14:01:19Z