cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Prototype998
by New Contributor III
  • 2480 Views
  • 5 replies
  • 2 kudos

Resolved! reading multiple csv files using pathos.multiprocessing

I'm using PySpark and Pathos to read numerous CSV files and create many DF, but I keep getting this problem.code for the same:-from pathos.multiprocessing import ProcessingPooldef readCsv(path):  return spark.read.csv(path,header=True)csv_file_list =...

dbx_error
  • 2480 Views
  • 5 replies
  • 2 kudos
Latest Reply
Prototype998
New Contributor III
  • 2 kudos

@Ajay Pandey​ @Rishabh Pandey​ 

  • 2 kudos
4 More Replies
Anonymous
by Not applicable
  • 504 Views
  • 0 replies
  • 0 kudos

Using multiple clouds

Are there recommendations and/or examples of leveraging AWS and Azure with Databricks? If so, is there any best practices to follow? Want to ensure we avoid expensive data transfer across clouds

  • 504 Views
  • 0 replies
  • 0 kudos
Labels