I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing to the topic and throwing an exception with code 40403 something. Is there an alternate way to do this?
error message:
Caused by: RestClientException: Schema not found; error code: 40403
This is how I am using to_avro:
val kafkaDf= df.select(to_avro($"value", lit("topicName-value"), schemaRegistry).as("value"))
kafkaDf
.write
.format("kafka")
.option("kafka.bootstrap.servers", bootstrapServer)
.option("topic", topicName)
.option("schema.registry.url", schemaRegistry)
.option(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistry)
.save()