Databricks

aladda · ‎06-23-2021

aladda · ‎06-23-2021

gzip format is not splittable so the load process is sequential and thus slower. You can either try to split the CSV up into parts, gzip those separately and load them. Alternatively bzip is a splittable zip format that is better to work with

Or you can also try Pigz alternative to gzip - https://ostechnix.com/pigz-compress-and-decompress-files-in-parallel-in-linux/

View solution in original post

aladda · ‎06-23-2021

gzip format is not splittable so the load process is sequential and thus slower. You can either try to split the CSV up into parts, gzip those separately and load them. Alternatively bzip is a splittable zip format that is better to work with

Or you can also try Pigz alternative to gzip - https://ostechnix.com/pigz-compress-and-decompress-files-in-parallel-in-linux/

Databricks

How can I speed up the loading of a large zipped CSV file in databricks

Announcing the General Availability of Databricks Asset Bundles

How to successfully build GenAI applications

Registration now open! Databricks Data + AI Summit 2024

Meet DBRX, the New Standard for High-Quality LLMs

Register now and save 50% on training at Data + AI Summit!