topic Re: Preferred compression format for ingesting large amounts of JSON files with Autoloader in Data Engineering

Preferred compression format for ingesting large amounts of JSON files with Autoloader

Volker — Mon, 29 Jan 2024 11:24:05 GMT

Hello Databricks Community,

in an IOT context we plan to ingest a large amount of JSON files (~2 Million per Day). The JSON files are in json lines format und need to be compressed on the IOT devices. We can provide suggestions for the type of compression that is optimal for ingesting these files.
The internet resources that we found suggest different compression formats that all have their pros and cons. We have currently looked at gzip and bzip2 compressions and it looks like bzip2 could be more performant than gzip.

Does anyone have experience with such a usecase and could provide some arguments in favor of a certain compression format or recommend other compression formats?

Thanks in advance!

Re: Preferred compression format for ingesting large amounts of JSON files with Autoloader

jose_gonzalez — Mon, 29 Jan 2024 23:43:22 GMT

Could you provide more details? for example, your source will be the JSON files, is your sink a Delta table? (assuming you will use auto loader to ingest your data) if that the case, then your Delta tables will be compressed already.

Re: Preferred compression format for ingesting large amounts of JSON files with Autoloader

Volker — Tue, 06 Feb 2024 09:13:09 GMT

Hi,

sorry I guess my response wasn't sent. The source are JSON files that are uploaded to an S3 bucket. The sink will be a Delta Table and we are using autoloader.
The question was about the compression format of the incoming JSON files, e.g. if it would be better to compress them using gzip or bzip2 or any other format. The compression ratio is not considered, it is just a matter of performance.

Thank you!

@jose_gonzalez