https://community.databricks.com/t5/data-engineering/autoloader-solution-for-binary-files/m-p/3890#M776 <P>We have solution implemented for ingesting binary file ( .ZIP ) into delta lake, Currently we are using the below solution within our pipeline.</P><P></P><OL><LI>Unzip the file and extract the XML file.</LI><LI>Parse the XML using python libraries.</LI><LI>Flatten the nested xml columns.</LI><LI>Store it to delta table.</LI></OL><P></P><P>This solution is working fine for small set of files ( 25 ). When we are processing large set of files ( 650 ) it is taking more time than expected. </P><P></P><P>Would like to know if we have a better solution to speed up the process.</P><P></P><P>Few things to note about the Xml file, This is a nested XML file which is having around 600 columns.</P><P></P> Tue, 30 May 2023 09:26:33 GMT Veeru245 2023-05-30T09:26:33Z https://community.databricks.com/t5/data-engineering/autoloader-solution-for-binary-files/m-p/3890#M776 <P>We have solution implemented for ingesting binary file ( .ZIP ) into delta lake, Currently we are using the below solution within our pipeline.</P><P></P><OL><LI>Unzip the file and extract the XML file.</LI><LI>Parse the XML using python libraries.</LI><LI>Flatten the nested xml columns.</LI><LI>Store it to delta table.</LI></OL><P></P><P>This solution is working fine for small set of files ( 25 ). When we are processing large set of files ( 650 ) it is taking more time than expected. </P><P></P><P>Would like to know if we have a better solution to speed up the process.</P><P></P><P>Few things to note about the Xml file, This is a nested XML file which is having around 600 columns.</P><P></P> Tue, 30 May 2023 09:26:33 GMT https://community.databricks.com/t5/data-engineering/autoloader-solution-for-binary-files/m-p/3890#M776 Veeru245 2023-05-30T09:26:33Z