This does not help solve the problem. I have the schema registered but somehow databricks's implementation of to_avro is not able to pick the correct one. I had to define my own Producer and Generic Avro records to send to the Kafka. to_avro utility method is not able to accomplish that.