cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
vishwanath_1
New Contributor III
since ‎01-11-2024
‎04-01-2024

User Stats

  • 7 Posts
  • 0 Solutions
  • 0 Kudos given
  • 0 Kudos received

User Activity

Loading spark dataframe to Mongo collection isn't allowing nulls

02-26-2024
I am using below command to push DataFrame to Mongo Collection.There are few null values in String and Double datatype columns , we see these are getting missed when pushed to mongo even after using the option("ignoreNullValues", false) inputproddata...

Need Suggestion for better caching strategy

01-22-2024
i have below steps to perform 1.Read a csv file (considerably huge file .. ~100gb)2.add index using zipwithindex function 3.repartition dataframe 4.Passing on to another function .Can you suggest the best optimized caching strategy to execute these c...

i am reading a 130gb csv file with multi line true it is taking 4 hours just to read

01-11-2024
reading 130gb file  without  multi line true it is 6 minutes my file has data in multi liner .How to speed up the reading time here .. i am using below commandInputDF=spark.read.option("delimiter","^").option("header",false).option("encoding","UTF-8"...