topic Re: i am reading a 130gb csv file with multi line true it is taking 4 hours just to read in Data Engineering

i am reading a 130gb csv file with multi line true it is taking 4 hours just to read

vishwanath_1 — Fri, 12 Jan 2024 03:06:25 GMT

reading 130gb file without multi line true it is 6 minutes

my file has data in multi liner .

How to speed up the reading time here ..

i am using below command

InputDF=spark.read.option("delimiter","^")

.option("header",false)

.option("encoding","UTF-8")

.option("multiLine","true")

option("quote","\"")

.option("escape","\"").csv(inputFileName)

Lakshay — Fri, 12 Jan 2024 13:02:11 GMT

Hi @vishwanath_1 , Can you try setting the below config if this resolves the issue?

set spark.databricks.sql.csv.edgeParserSplittable=true;

vishwanath_1 — Mon, 22 Jan 2024 06:18:53 GMT

By using set spark.databricks.sql.csv.edgeParserSplittable=true;

There is now taking 30 mins lesser time than usual 4 hours.

Any other setting which can be used to make it faster?

Lakshay — Mon, 22 Jan 2024 16:00:11 GMT

You can also try using Photon. That can also help speed up the read operation.

subash_07 — Sun, 22 Sep 2024 07:43:43 GMT

Hi @Lakshay , where did you find this config ? can you give link ?