Got the same error with lower DBR.

I am using OAuth user-to-machine (U2M) authentication and a configuration profile named DEFAULT which contains host, clusterId and auth_type saved in .databrickscfg file. I logged in using databricks cli before running the code.

To create SparkSession I used, 

 val spark = DatabricksSession.builder().remote().getOrCreate()

On debugging I could see that the, cluster_id ,host ,sdkConfig,token is None.

 I followed this tutorial, https://docs.databricks.com/en/dev-tools/databricks-connect/scala/index.html#tutorial