topic Re: I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing t... in Data Engineering https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12439#M7239 <P>This does not help solve the problem. I have the schema registered but somehow databricks's implementation of to_avro is not able to pick the correct one. I had to define my own Producer and Generic Avro records to send to the Kafka. to_avro utility method is not able to accomplish that.</P> Fri, 13 Jan 2023 20:30:44 GMT dheeraj2444 2023-01-13T20:30:44Z I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing t... https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12437#M7237 <P>I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing to the topic and throwing an exception with code 40403 something. Is there an alternate way to do this?</P><P></P><P>error message:</P><PRE><CODE>Caused by: RestClientException: Schema not found; error code: 40403</CODE></PRE><P>This is how I am using to_avro:</P><PRE><CODE>val kafkaDf= df.select(to_avro($"value", lit("topicName-value"), schemaRegistry).as("value")) kafkaDf .write .format("kafka") .option("kafka.bootstrap.servers", bootstrapServer) .option("topic", topicName) .option("schema.registry.url", schemaRegistry) .option(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistry) .save()</CODE></PRE><P></P><P></P> Thu, 12 Jan 2023 03:53:04 GMT https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12437#M7237 dheeraj2444 2023-01-12T03:53:04Z Re: I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing t... https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12439#M7239 <P>This does not help solve the problem. I have the schema registered but somehow databricks's implementation of to_avro is not able to pick the correct one. I had to define my own Producer and Generic Avro records to send to the Kafka. to_avro utility method is not able to accomplish that.</P> Fri, 13 Jan 2023 20:30:44 GMT https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12439#M7239 dheeraj2444 2023-01-13T20:30:44Z Re: I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing t... https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12438#M7238 <P>Hi,</P><P>Could you please refer to <A href="https://github.com/confluentinc/kafka-connect-elasticsearch/issues/59" target="test_blank">https://github.com/confluentinc/kafka-connect-elasticsearch/issues/59</A> and let us know if this helps.</P><P></P> Thu, 12 Jan 2023 22:16:13 GMT https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12438#M7238 Debayan 2023-01-12T22:16:13Z Re: I am trying to write a data frame to Kafka topic with Avro schema for key and value using a schema registry URL. The to_avro function is not writing t... https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12440#M7240 <P>@Dheeraj Singh​&nbsp;, Did it solve the issue? If yes, could you please log a case with Databricks to look into the implementation?</P> Mon, 16 Jan 2023 04:42:50 GMT https://community.databricks.com/t5/data-engineering/i-am-trying-to-write-a-data-frame-to-kafka-topic-with-avro/m-p/12440#M7240 Debayan 2023-01-16T04:42:50Z