Hi @Vasu_Kumar_T 

This is a classic Spark performance inconsistency issue. The fact that it works fine in your notebook
but degrades after deployment suggests several potential causes. Here are the most likely culprits and solutions:

Primary Suspects
1. Data Skew Variations
Your salt-based repartitioning might not be consistently effective
if the underlying data distribution changes between runs or environments.

2. Cluster Resource Allocation
Fixed 20 nodes doesn't guarantee same resource allocation as auto-scaling.

3. Memory and Executor Configuration

Environment-Specific Solutions
Check these deployment differences:
- Spark version consistency between notebook and deployment
- Network bandwidth between nodes in production vs. development
- Storage type (SSD vs. HDD) and I/O throughput