Apply Avro defaults when writing to Confluent Kafka
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2022 09:45 AM
I have an avro schema for my Kafka topic. In that schema it has defaults. I would like to exclude the defaulted columns from databricks and just let them default as an empty array.
Sample avro, trying to not provide the UserFields because I can't store an empty array in Delta (Parquet) in databricks and just let it default in as empty.
{
"name": "UserFields",
"type": {
"type": "array",
"items": {
"name" : "ud_record",
"type" : "record",
"fields": [{
"name": "userFieldName",
"type": "string"
},{
"name": "userFieldValue",
"type": ["null", "string"]
}]
},
},
"default": []
}
Labels:
- Labels:
-
Avro