I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing t...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2023 07:53 PM
I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing to the topic and throwing an exception with code 40403 something. Is there an alternate way to do this?
error message:
Caused by: RestClientException: Schema not found; error code: 40403
This is how I am using to_avro:
val kafkaDf= df.select(to_avro($"value", lit("topicName-value"), schemaRegistry).as("value"))
kafkaDf
.write
.format("kafka")
.option("kafka.bootstrap.servers", bootstrapServer)
.option("topic", topicName)
.option("schema.registry.url", schemaRegistry)
.option(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistry)
.save()
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2023 02:16 PM
Hi,
Could you please refer to https://github.com/confluentinc/kafka-connect-elasticsearch/issues/59 and let us know if this helps.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2023 12:30 PM
This does not help solve the problem. I have the schema registered but somehow databricks's implementation of to_avro is not able to pick the correct one. I had to define my own Producer and Generic Avro records to send to the Kafka. to_avro utility method is not able to accomplish that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-15-2023 08:42 PM
@Dheeraj Singh , Did it solve the issue? If yes, could you please log a case with Databricks to look into the implementation?
![](/skins/images/582998B45490C7019731A5B3A872C751/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/582998B45490C7019731A5B3A872C751/responsive_peak/images/icon_anonymous_message.png)