cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Handling Kafka Topics with Avro Schema

leelee3000
New Contributor III
New Contributor III

Our input data resides in a Kafka topic, and we utilize the Kafka schema registry with Avro schemas. While I can retrieve the schema from the registry, I am facing challenges creating a Spark DataFrame that correctly serializes data for streaming reads. Could you provide guidance on efficiently using Avro schemas from the schema registry to create Spark DataFrames for streaming reads? The aim would be to have this functionality modular as possible. I want be able to parameterize the function/notebook and pass the topic and the schema on the registry and have it create the stream that reads it and serializes it correctly into a delta live table.

0 REPLIES 0
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!