topic Re: reading multiple csv files using pathos.multiprocessing in Data Engineering

reading multiple csv files using pathos.multiprocessing

Prototype998 — Thu, 14 Jul 2022 06:51:20 GMT

I'm using PySpark and Pathos to read numerous CSV files and create many DF, but I keep getting this problem.

code for the same:-

from pathos.multiprocessing import ProcessingPool

def readCsv(path):

return spark.read.csv(path,header=True)

csv_file_list = [file[0][5:] for file in dbutils.fs.ls("/databricks-datasets/COVID/coronavirusdataset/") if file[1].endswith(".csv")]

pool = ProcessingPool(2)

results = pool.map(readCsv, csv_file_list)

Re: reading multiple csv files using pathos.multiprocessing

AmanSehgal — Thu, 14 Jul 2022 09:41:06 GMT

You actually don't need to filter `.csv` files like that.

You can use `pathGlobFilter` to do a regex match for selecting files that matches provided regular expression.

df = spark.read.option("pathGlobFilter","*.csv").csv(upload_path)

Re: reading multiple csv files using pathos.multiprocessing

Vidula — Sun, 04 Sep 2022 07:00:49 GMT

Hi @Punit Chauhan

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: reading multiple csv files using pathos.multiprocessing

Prototype998 — Thu, 22 Dec 2022 09:30:41 GMT

@Ajay Pandey @Rishabh Pandey

Re: reading multiple csv files using pathos.multiprocessing

Rishabh-Pandey — Thu, 22 Dec 2022 09:34:10 GMT

hey @Punit Chauhan refer this code

from multiprocessing.pool import ThreadPool
pool = ThreadPool(5)
notebooks = ['dim_1', 'dim_2']
pool.map(lambda path: dbutils.notebook.run("/Test/Threading/"+path, timeout_seconds= 60, arguments={"input-data": path}),notebooks)

Re: reading multiple csv files using pathos.multiprocessing

Prototype998 — Thu, 22 Dec 2022 09:35:14 GMT

thanks @Rishabh Pandey