Hi, Could you please confirm the usage of your cluster while running this job? you can monitor the performance here: https://docs.databricks.com/clusters/clusters-manage.html#monitor-performance with different metrics.
Also, please tag @Debayan with your next response which will notify me. Thank you!
