cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to resolve "java.lang.ClassNotFoundException: com.databricks.spark.util.RegexBasedAWSSecretKeyRedactor" when running Scala Spark project using databricks-connect ?

Databach
New Contributor

Currently I am learning how to use databricks-connect to develop Scala code using IDE (VS Code) locally. The set-up of the databricks-connect as described here https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect was succuesful. I can compile PySpark scripts with no problem. However, I struggle to run the scala example as provided in the Scala demo object. I receive the java.lang.ClassNotFoundException:

imageMy build.sbt follows the suggested pattern:

build.sbtDo you have any suggestion on what I am missing ? The error occurs when building the SparkSession.

val spark = SparkSession.builder.master("local").getOrCreate()

A 'hello world' prints succesfully.

Note: I am using Databricks on Azure

0 REPLIES 0
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.