Showing results for 
Search instead for 
Did you mean: 
New Contributor III
since ‎01-11-2024

User Stats

  • 7 Posts
  • 0 Solutions
  • 0 Kudos given
  • 0 Kudos received

User Activity

I am using below command to push DataFrame to Mongo Collection.There are few null values in String and Double datatype columns , we see these are getting missed when pushed to mongo even after using the option("ignoreNullValues", false) inputproddata...
i have below steps to perform 1.Read a csv file (considerably huge file .. ~100gb)2.add index using zipwithindex function 3.repartition dataframe 4.Passing on to another function .Can you suggest the best optimized caching strategy to execute these c...
reading 130gb file  without  multi line true it is 6 minutes my file has data in multi liner .How to speed up the reading time here .. i am using below"delimiter","^").option("header",false).option("encoding","UTF-8"...