Mo
Databricks Employee
Databricks Employee

hey @AlokThampi,

it's difficult to understand what's going on here (not having access to spark UI, query profile or any idea about the dimensions of these tables) but you can give these a try:

  1. make sure the keys have the same data type (i.e. all three are Long or INT)
  2. you are using 3 tables, while I see you optimized only two of them, and reorganized the data layout using zorder.I highly suggest to use liquid clustering on all three
    1. the ReportTable should be clustered by ORDER_ID.
    2. the EKBETable should be clustered by BELNR
    3.  the PurchaseOrder should be clustered by PO_NO.

Let me know if these have any effect or please provide more details to be able to find the bottleneck of this join 😉