Can you try to run a test with maximum simplified spark_connect (so just method and spark_home).

Additionally please check following:

• Only the following Databricks Runtime versions are supported:
  • Databricks Runtime 9.1 LTS ML, Databricks Runtime 9.1 LTS
  • Databricks Runtime 7.3 LTS ML, Databricks Runtime 7.3 LTS
  • Databricks Runtime 6.4 ML, Databricks Runtime 6.4
  • Databricks Runtime 5.5 LTS ML, Databricks Runtime 5.5 LTS
• The minor version of your client Python installation must be the same as the minor Python version of your Azure Databricks cluster. The table shows the Python version installed with each Databricks Runtime.