https://community.databricks.com/t5/data-engineering/how-to-write-data-to-confluent-kafka-with-schemaregistry-format/m-p/73080#M34638 <P>Hi There!<BR />I am to trying write a batch data to kafka topic with schema registry in databricks using pyspark, i serialize the data with pyspark to_avro function and write it to the topic, but the consumers can’t read the schema id. If they do not separate the schema_id first 5 bytes they can read the data as well.<BR />I read the avro schema from .avsc file downloaded in confluent, and it’s version is ok.</P><P>This is my script:</P><P>df.select(to_avro(carrossel_schema, schema_str)).alias("value") \<BR />.write \<BR />.format('kafka') \<BR />.option("kafka.bootstrap.servers", confluent_server) \<BR />.option("topic", topico) \<BR />.option("kafka.security.protocol", "SASL_SSL") \<BR />.option(<BR />"kafka.sasl.jaas.config",<BR />f"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{confluent_user}' password='{confluent_pass}';"<BR />) \<BR />.option("kafka.ssl.endpoint.identification.algorithm", "https") \<BR />.option("kafka.sasl.mechanism", "PLAIN") \<BR />.save()</P><P><BR />return df</P> Wed, 12 Jun 2024 20:01:41 GMT GCosta 2024-06-12T20:01:41Z https://community.databricks.com/t5/data-engineering/how-to-write-data-to-confluent-kafka-with-schemaregistry-format/m-p/73080#M34638 <P>Hi There!<BR />I am to trying write a batch data to kafka topic with schema registry in databricks using pyspark, i serialize the data with pyspark to_avro function and write it to the topic, but the consumers can’t read the schema id. If they do not separate the schema_id first 5 bytes they can read the data as well.<BR />I read the avro schema from .avsc file downloaded in confluent, and it’s version is ok.</P><P>This is my script:</P><P>df.select(to_avro(carrossel_schema, schema_str)).alias("value") \<BR />.write \<BR />.format('kafka') \<BR />.option("kafka.bootstrap.servers", confluent_server) \<BR />.option("topic", topico) \<BR />.option("kafka.security.protocol", "SASL_SSL") \<BR />.option(<BR />"kafka.sasl.jaas.config",<BR />f"kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{confluent_user}' password='{confluent_pass}';"<BR />) \<BR />.option("kafka.ssl.endpoint.identification.algorithm", "https") \<BR />.option("kafka.sasl.mechanism", "PLAIN") \<BR />.save()</P><P><BR />return df</P> Wed, 12 Jun 2024 20:01:41 GMT https://community.databricks.com/t5/data-engineering/how-to-write-data-to-confluent-kafka-with-schemaregistry-format/m-p/73080#M34638 GCosta 2024-06-12T20:01:41Z