- 4079 Views
- 5 replies
- 2 kudos
Resolved! reading multiple csv files using pathos.multiprocessing
I'm using PySpark and Pathos to read numerous CSV files and create many DF, but I keep getting this problem.code for the same:-from pathos.multiprocessing import ProcessingPooldef readCsv(path): return spark.read.csv(path,header=True)csv_file_list =...
- 4079 Views
- 5 replies
- 2 kudos
Latest Reply
- 806 Views
- 0 replies
- 0 kudos
Using multiple clouds
Are there recommendations and/or examples of leveraging AWS and Azure with Databricks? If so, is there any best practices to follow? Want to ensure we avoid expensive data transfer across clouds
- 806 Views
- 0 replies
- 0 kudos