cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

what are the join hints, available in spark 3.0, and how does it help compare to pervious spark version

User16826994223
Honored Contributor III

what are the join hints, available in spark 3.0, and how does it help compare to pervious spark version 

1 ACCEPTED SOLUTION

Accepted Solutions

Srikanth_Gupta_
Valued Contributor

4 types of join hints in Spark 3.0

BROADCAST

MERGE

SHUFFLE_HASH

SHUFFLE_REPLICATE_NL

May be good idea to enable Adaptive Query Execution which speeds up Spark SQL join during run time

In Spark 3.0, Adaptive Query Execution comes with below features

Dynamically coalescing shuffle partitions

Dynamically switching join strategies

Dynamically optimizing skew joins

more details on ADE are available here 

View solution in original post

1 REPLY 1

Srikanth_Gupta_
Valued Contributor

4 types of join hints in Spark 3.0

BROADCAST

MERGE

SHUFFLE_HASH

SHUFFLE_REPLICATE_NL

May be good idea to enable Adaptive Query Execution which speeds up Spark SQL join during run time

In Spark 3.0, Adaptive Query Execution comes with below features

Dynamically coalescing shuffle partitions

Dynamically switching join strategies

Dynamically optimizing skew joins

more details on ADE are available here 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.