cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
ssaloni_bhatia
by • New Contributor
  • 682 Views
  • 1 replies
  • 0 kudos

Spark read not working in parallel

Hi All,Am trying to run a spark jdbc query and the read is running on only 1 worker and is taking a lot of time. Usually, it takes 4 minutes in aws lambda but now it is taking approx 10 minutes while using databricks. There are only 4m records (write...

Data Engineering
Reply
  • 682 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Saloni Bhatia​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
tanjil
by • New Contributor III
  • 3427 Views
  • 6 replies
  • 6 kudos

Resolved! Read and transform CSVs in parallel.

I need to read and transform several CSV files and then append them to a single data frame. I am able to do this in databricks using simple for loops, but I would like to speed this up.Below is the rough structure of my code: for filepath in all_file...

Data Engineering
Reply
  • 3427 Views
  • 6 replies
  • 6 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 6 kudos

Hi @tanjil​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 6 kudos
5 More Replies
Labels