cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
brickster_2018
by • Databricks Employee
  • 8731 Views
  • 2 replies
  • 0 kudos

Resolved! What is the maximum limit of data that can be broadcasted using broadcast join

Data Engineering
Reply
  • 8731 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
lchari
New Contributor II
  • 0 kudos

Is the limit per "table/dataframe" or for all tables/dataframes put together?The driver collects the data from all executors (which are having the respective table or dataframe) and distributes to all executors. When will the memory be released in bo...

  • 0 kudos
1 More Replies
Chalki
by • New Contributor III
  • 3401 Views
  • 2 replies
  • 4 kudos

Resolved! Delta Table Merge statement is not accepting broadcast hint

I have a statement like this with pyspark:target_tbl.alias("target")\            .merge(stage_df.hint("broadcast").alias("source"), merge_join_expr)\                .whenMatchedUpdateAll()\                .whenNotMatchedInsertAll()\                .w...

Data Engineering
Reply
  • 3401 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Nikolay Chalkanov​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 4 kudos
1 More Replies
gauthamchettiar
by • New Contributor II
  • 1771 Views
  • 0 replies
  • 1 kudos

Spark always performing broad casts irrespective of spark.sql.autoBroadcastJoinThreshold during streaming merge operation with DeltaTable.

I am trying to do a streaming merge between delta tables using this guide - https://docs.delta.io/latest/delta-update.html#upsert-from-streaming-queries-using-foreachbatchOur Code Sample (Java): Dataset<Row> sourceDf = sparkSession ...

BroadCastJoin 1M
Data Engineering
Reply
  • 1771 Views
  • 0 replies
  • 1 kudos
jose_gonzalez
by • Databricks Employee
  • 2658 Views
  • 1 replies
  • 1 kudos

Resolved! Are there any limitations on my broadcast joins?

I would like to know if there are any broadcast joins limitations.

Data Engineering
Reply
  • 2658 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Yes, there are a couple limitation. Please find below the details:> It will not perform broadcast join if the table has 512 million or more rows > It will not perform broadcast join if the table is larger than 8GB

  • 1 kudos
Labels