I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing t...

[dheeraj2444]

I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing to the topic and throwing an exception with code 40403 something. Is there an alternate way to do this?

error message:

Caused by: RestClientException: Schema not found; error code: 40403

This is how I am using to_avro:

val kafkaDf= df.select(to_avro($"value", lit("topicName-value"), schemaRegistry).as("value"))
kafkaDf
  .write
  .format("kafka")
  .option("kafka.bootstrap.servers", bootstrapServer)
  .option("topic", topicName)
  .option("schema.registry.url", schemaRegistry)
  .option(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistry)
  .save()

[Debayan]

Hi,

Could you please refer to https://github.com/confluentinc/kafka-connect-elasticsearch/issues/59 and let us know if this helps.

[dheeraj2444]

This does not help solve the problem. I have the schema registered but somehow databricks's implementation of to_avro is not able to pick the correct one. I had to define my own Producer and Generic Avro records to send to the Kafka. to_avro utility method is not able to accomplish that.

[Debayan]

@Dheeraj Singh​ , Did it solve the issue? If yes, could you please log a case with Databricks to look into the implementation?