cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Chalki
by New Contributor III
  • 1590 Views
  • 2 replies
  • 4 kudos

Resolved! Delta Table Merge statement is not accepting broadcast hint

I have a statement like this with pyspark:target_tbl.alias("target")\            .merge(stage_df.hint("broadcast").alias("source"), merge_join_expr)\                .whenMatchedUpdateAll()\                .whenNotMatchedInsertAll()\                .w...

  • 1590 Views
  • 2 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Nikolay Chalkanov​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 4 kudos
1 More Replies
gauthamchettiar
by New Contributor II
  • 785 Views
  • 0 replies
  • 1 kudos

Spark always performing broad casts irrespective of spark.sql.autoBroadcastJoinThreshold during streaming merge operation with DeltaTable.

I am trying to do a streaming merge between delta tables using this guide - https://docs.delta.io/latest/delta-update.html#upsert-from-streaming-queries-using-foreachbatchOur Code Sample (Java): Dataset<Row> sourceDf = sparkSession ...

BroadCastJoin 1M
  • 785 Views
  • 0 replies
  • 1 kudos
User16869510359
by Esteemed Contributor
  • 4516 Views
  • 1 replies
  • 0 kudos
  • 4516 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

By default, only 10 MB of data can be broadcasted. spark.sql.autoBroadcastJoinThreshold can be increased up to 8GBThere is an upper limit in terms of records as well. We can't broadcast more than 512m records. So its either 512m records or 8GB which...

  • 0 kudos
jose_gonzalez
by Moderator
  • 1220 Views
  • 1 replies
  • 1 kudos

Resolved! Are there any limitations on my broadcast joins?

I would like to know if there are any broadcast joins limitations.

  • 1220 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Yes, there are a couple limitation. Please find below the details:> It will not perform broadcast join if the table has 512 million or more rows > It will not perform broadcast join if the table is larger than 8GB

  • 1 kudos
Labels