Hi Ravi,

Thanks for the reply with the useful links. Correct me if I am wrong. I am thinking to create a new cluster and attach the job to it. During this process I will be choosing the new 9.1 LTS and it automatically picks up the Spark 3.1.2. and new Python 3.8.8 environment. Then run the code in dev to check the potential errors and fix them. is that right? or anything different?

Thanks in adavnce.

Kiran