Accessing confluent schema registry from databricks with scala fails with 401 (just for scala, not python, just in databricks)

alonisser — Sat, 16 Oct 2021 16:36:27 GMT

Nore, I've tested with the same connection variable:

locally with scala - works (via the same prod schema registry)
in the cluster with python - works
in the cluster with scala - fails with 401 auth error

def setupSchemaRegistry(schemaRegistryUrl: String, confluentRegistryApiKey: String, confluentRegistrySecret: String): CachedSchemaRegistryClient = {
 
   val props = Map(
      "basic.auth.credentials.source" -> "USER_INFO",
      "basic.auth.user.info" -> s"$confluentRegistryApiKey:$confluentRegistrySecret",
      "schema.registry.basic.auth.credentials.source" -> "USER_INFO", //tried both this version just to be sure
      "schema.registry.url" -> schemaRegistryUrl,
      "schema.registry.basic.auth.user.info" -> s"$confluentRegistryApiKey:$confluentRegistrySecret"
    ).asJava  
//   val restService = new RestService(schemaRegistryUrl)
    
 
    // The format of the schema registry credentials below is confluentRegistryApiKey:confluentRegistrySecret
    println(s"schema registry info $schemaRegistryUrl $confluentRegistryApiKey $confluentRegistrySecret")
    // also tried the version where I pass the restService inside instead, etc
    new CachedSchemaRegistryClient(schemaRegistryUrl, 100, props)
}
val schemaRegistry = setupSchemaRegistry(schemaRegistryUrl, confluentRegistryApiKey, confluentRegistrySecret)
val res = schemaRegistry.getSchemaById(number) //gets 401

databricks runtime : 9.1

confluent client 6.2.1

Why would it fail just in scala, just from databricks cluster? I can't explain that!

I'll be glad for any help

Re: Accessing confluent schema registry from databricks with scala fails with 401 (just for scala, not python, just in databricks)

Anonymous — Sun, 17 Oct 2021 18:57:44 GMT

Hi @Alon Nisser - My name is Piper and I'm one of the community moderators. Welcome and thanks for your question. Let's give this a bit longer to see what the community says.

Re: Accessing confluent schema registry from databricks with scala fails with 401 (just for scala, not python, just in databricks)

alonisser — Mon, 18 Oct 2021 11:03:31 GMT

Found the issue: it's the uber package mangling some dependency resolving, which I fixed

Another issue, is that currently you can't use 6.* branch of confluent schema registry client in databricks, because the avro version is different then the one supported in spark 3.1

topic Re: Accessing confluent schema registry from databricks with scala fails with 401 (just for scala, not python, just in databricks) in Data Engineering

Accessing confluent schema registry from databricks with scala fails with 401 (just for scala, not python, just in databricks)

Re: Accessing confluent schema registry from databricks with scala fails with 401 (just for scala, not python, just in databricks)

Re: Accessing confluent schema registry from databricks with scala fails with 401 (just for scala, not python, just in databricks)