Thanks for the update @Dan Zafar​ .

Just ran the job again and still seeing spark version 3.1.0. Should I be using spark32 (or something similar) when invoking spark session for me to pick up the correct spark version?

Any ETA on the spark 3.2 version availability will be great.

Thanks