The code works fine on databricks cluster, but this code is part of a unit test in local env. then submitted to a branch->PR->merged into master branch.

Thanks for the advice on using DBX. I will give DBX a try again even though I've already tried.

I'll be heads up for Spark Connect.

Thank you.