I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing to the topic and throwing an exception with code 40403 something. Is there an alternate way to do this?

error message:

Caused by: RestClientException: Schema not found; error code: 40403

This is how I am using to_avro:

val kafkaDf= df.select(to_avro($"value", lit("topicName-value"), schemaRegistry).as("value"))
kafkaDf
  .write
  .format("kafka")
  .option("kafka.bootstrap.servers", bootstrapServer)
  .option("topic", topicName)
  .option("schema.registry.url", schemaRegistry)
  .option(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistry)
  .save()