Flatfiles ingestion on Bronze layer, 'to schema' or 'not to schemarize'?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-08-2022 05:32 AM
Hi all, What is the general guideline for handling flatfiles (xml, json with several nested hierarchies that is also schema evolving) in the bronze layer?
Should I persist the file content into a single column as text in the parquet file
or
should I let spark infer a schema and have it output a parquet file with several columns representing the content of the xml/json file?
Labels:
- Labels:
-
Bronze Layer
-
Medallion Architecture