Databricks Community

To get closer to the error:There is same mystical size limit.

Hi @reachrishav ,Since 14.3 there is a native support for read and write XML files. Maybe check if it works faster than the library that you've used:Read and write XML files | Databricks on AWSAnd you've mentioned that you write python function to fl...

Hi @YS1 ,Have you added the python file in the Pipeline settings, in the list of source code?     

One other thought -- if you are considering using pandas_udf api, there is a way to control batch size there:pandas_udf guide   note the comments there about arrow batch size params.

If were were to upgrade to ADLSg2, but retain the same structure, would there be scope for this method above to be improved (besides moving to notification mode)?

Hi @hpant would you consider testing the new VARIANT type for your JSON data? I appreciate it will require rewriting the next step in your pipeline, but should be more robust wrt errors.  Disclaimer: I haven't personally tested variant with Autoloade...