@nengen  Try using EXPLAIN EXTENDED: This provides a detailed breakdown of the logical and physical plan of a query in Spark SQL.

Based on the EXPLAIN EXTENDED output, here are a few things to consider:

• Broadcast Exchange: If the join causes data skew, consider switching to a sort-merge join.
• FileScan: If the scan is slow, consider partitioning or caching the data to improve performance.
• Filter Pushdown: Ensure the most restrictive filters are applied early to reduce the amount of data processed.

Please review for more details