add custom metadata to avro file with pyspark
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2023 11:03 AM
Hello,
i need to add a custom metadata into a avro file. The avro file containt data.
we have tried to use "option" within the write function but it's not taken without generated any error.
df.write.format("avro").option("avro.codec", "snappy").option("header", "metadata_key:metadata_value").mode("overwrite").save("/tmp/avro_with_metadata")
I'm seeking a solution to add a custom metadata into a data avro file.
Thanks,
Zakaria
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-24-2023 09:08 AM
thank you a lot for your answer, it's very helpful.
I have a additionnal question please, is it possible to add a avro binnary metada inside the json avro_schema?