How to resolve "java.lang.ClassNotFoundException: com.databricks.spark.util.RegexBasedAWSSecretKeyRedactor" when running Scala Spark project using databricks-connect ?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-24-2022 02:34 AM
Currently I am learning how to use databricks-connect to develop Scala code using IDE (VS Code) locally. The set-up of the databricks-connect as described here https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect was succuesful. I can compile PySpark scripts with no problem. However, I struggle to run the scala example as provided in the Scala demo object. I receive the java.lang.ClassNotFoundException:
My build.sbt follows the suggested pattern:
Do you have any suggestion on what I am missing ? The error occurs when building the SparkSession.
val spark = SparkSession.builder.master("local").getOrCreate()A 'hello world' prints succesfully.
Note: I am using Databricks on Azure
Labels: