Hi @SDas1 , Based on the provided information, the error message "Driver of the cluster (0307-***-gpbwt) was restarted during the run" indicates that the driver node of your cluster has restarted during the job run.
This could be due to a variety of reasons such as resource constraints, hardware failures or software issues.
For the "effectiveIntegrationRuntime": "vnet-ir-*-**** (Clark and Sons) error, it seems like there might be an issue with the integration runtime during the merge operation.
The inconsistencies in these errors could be due to varying conditions in your environment.
To troubleshoot these issues:
- Investigate the driver log and node logs of the driver node. They may provide insights into what caused the driver process to restart.
- Check if there are any custom spark configurations and libraries that could be causing these issues.
- For the integration runtime error, verify the configurations and ensure they are correct.
Please note that without more specific details about your environment and the exact operations you're performing, it's hard to provide a more precise solution.
