First, our process was triggered by the datafactory. First the connexion was set with token access, then with managed service identity.
We prove the untimely time out was not due to the datafactory by running directly the notebook.
Secondly, we tried running the notebook alone and scale up the inactivity time before shutdown.
But none of our tries lead to a success, we have had to scale up to 16 workers and restart every hour our pipeline to success. We did this for our integration plan but we refuse to face it again for our production plan.
Could you please, suggest to us some good advises.
You will find attached the executing code
Kind Regards,
Natacha
