Loading spark dataframe to Mongo collection isn't allowing nulls

vishwanath_1 — Tue, 27 Feb 2024 03:14:01 GMT

I am using below command to push DataFrame to Mongo Collection.

There are few null values in String and Double datatype columns , we see these are getting missed when pushed to mongo even after using the option("ignoreNullValues", false)

inputproddata.write.format("mongo").option("spark.mongodb.output.uri", connectionString).option("database",database).option("collection",collection).option("ignoreNullValues", false).mode("append").save()

Please let me know why this option does not work and suggest for any alternative solution

Re: Loading spark dataframe to Mongo collection isn't allowing nulls

vishwanath_1 — Thu, 14 Mar 2024 12:11:05 GMT

First approach works.. Thanks

topic Re: Loading spark dataframe to Mongo collection isn't allowing nulls in Data Engineering

Loading spark dataframe to Mongo collection isn't allowing nulls

Re: Loading spark dataframe to Mongo collection isn't allowing nulls