topic Re: Parallel processing of json files in databricks pyspark in Data Engineering

Parallel processing of json files in databricks pyspark

AzureDatabricks — Mon, 22 Nov 2021 07:34:20 GMT

How we can read files from azure blob storage and process parallel in databricks using pyspark.

As of now we are reading all 10 files at a time into dataframe and flattening it.

Thanks & Regards,

Sujata

Re: Parallel processing of json files in databricks pyspark

-werners- — Mon, 22 Nov 2021 07:49:18 GMT

if you use the spark json reader, it will happen in parallel automatically.

Depending on the cluster size, you will be able to read more files in parallel.

Mind that json usually are small files. Spark does not like a lot of small files, so performance may suffer.

Depending on the use case it can be a good idea to do an initial conversion to parquet/delta lake (which will take some time because of multiple small files), and then keep on adding new files to this table.

For your data jobs, you can read the parquet/delta lake which will be a lot faster.

Re: Parallel processing of json files in databricks pyspark

AzureDatabricks — Mon, 22 Nov 2021 09:51:27 GMT

can you provide us sample to read read json files parallel from blob. We are reading all files one by one from directory it is taking time to load into data frame

Thank you

Re: Parallel processing of json files in databricks pyspark

-werners- — Mon, 22 Nov 2021 09:54:07 GMT

spark.read.json("/mnt/dbfs/<ENTER PATH OF JSON DIR HERE>/*.json

you first have to mount your blob storage to databricks, I assume that is already done.

https://spark.apache.org/docs/latest/sql-data-sources-json.html

Re: Parallel processing of json files in databricks pyspark

SailajaB — Mon, 22 Nov 2021 10:59:03 GMT

Thank you.. We are using mount already..

Re: Parallel processing of json files in databricks pyspark

jose_gonzalez — Mon, 22 Nov 2021 19:57:46 GMT

Hi @Sailaja B ,

Check the number of stages and task when you are reading the JSON files. How many do you see? are you JSON files nested? how long does it takes to read a single JSON file?