https://community.databricks.com/t5/data-engineering/flatfiles-ingestion-on-bronze-layer-to-schema-or-not-to/m-p/23195#M15978 <P>Hi all, What is the general guideline for handling flatfiles (xml, json with several nested hierarchies that is also schema evolving) in the bronze layer?</P><P></P><P>Should I persist the file content into a single column as text in the parquet file </P><P></P><P>or </P><P></P><P>should I let spark infer a schema and have it output a parquet file with several columns representing the content of the xml/json file?</P> Tue, 08 Nov 2022 13:32:52 GMT 7effrey 2022-11-08T13:32:52Z https://community.databricks.com/t5/data-engineering/flatfiles-ingestion-on-bronze-layer-to-schema-or-not-to/m-p/23195#M15978 <P>Hi all, What is the general guideline for handling flatfiles (xml, json with several nested hierarchies that is also schema evolving) in the bronze layer?</P><P></P><P>Should I persist the file content into a single column as text in the parquet file </P><P></P><P>or </P><P></P><P>should I let spark infer a schema and have it output a parquet file with several columns representing the content of the xml/json file?</P> Tue, 08 Nov 2022 13:32:52 GMT https://community.databricks.com/t5/data-engineering/flatfiles-ingestion-on-bronze-layer-to-schema-or-not-to/m-p/23195#M15978 7effrey 2022-11-08T13:32:52Z