If you want to use PySpark UDFs, it’s important that your development machine’s installed minor version of Python match the minor version of Python that is included with Databricks Runtime installed on the cluster.

Please refer to the document and check if your setup meets the required configuration. Databricks Connect | Databricks on AWS

For Databricks Runtime 13.0 and higher, Databricks Connect is now built on open-source Spark Connect.