How to resolve "java.lang.ClassNotFoundException: com.databricks.spark.util.RegexBasedAWSSecretKeyRedactor" when running Scala Spark project using databricks-connect ?

Databach
New Contributor

Currently I am learning how to use databricks-connect to develop Scala code using IDE (VS Code) locally. The set-up of the databricks-connect as described here https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect was succuesful. I can compile PySpark scripts with no problem. However, I struggle to run the scala example as provided in the Scala demo object. I receive the java.lang.ClassNotFoundException:

imageMy build.sbt follows the suggested pattern:

build.sbtDo you have any suggestion on what I am missing ? The error occurs when building the SparkSession.

val spark = SparkSession.builder.master("local").getOrCreate()

A 'hello world' prints succesfully.

Note: I am using Databricks on Azure