topic Small json files issue . taking 2 hours to read 3000 files in Data Engineering

Small json files issue . taking 2 hours to read 3000 files

Subhasis — Wed, 13 Nov 2024 07:39:29 GMT

Hello I am trying to read 3000 json files which has only one records. It is taking 2 hours to read all the files . How can I perform this operation faster pls suggest.

Re: Small json files issue . taking 2 hours to read 3000 files

Subhasis — Wed, 13 Nov 2024 08:05:47 GMT

This is the code ---df1 = spark.read.format("json").options(inferSchema="true", multiLine="true").load(file1)

Re: Small json files issue . taking 2 hours to read 3000 files

szymon_dybczak — Wed, 13 Nov 2024 11:30:43 GMT

Hi @Subhasis

You can start off by specyfying schema upfront instead of using infer schema option. But to be honest, it is classical "small file problem". The best approach you can take is to compact those small files into larges ones.
Or you can read all them and save them as a parquet files with a proper partition size.
Take a look at below threads for inspiration:

Big data [Spark] and its small files problem – Garren's [Big] Data Blog

apache spark - Reading Millions of Small JSON Files from S3 Bucket in PySpark Very Slow - Stack Overflow