saurabh18cs
Honored Contributor III

‎02-10-2026 02:50 AM

Increasing samplesize to a 100000 forces the Mongo Spark Connector to pull huge amounts of documents into the Spark driver, which overwhelms memory and causes the driver JVM to die and restart.
 
1) I would suggest you not to rely on schema inference which is more intensive operation when you have 100+ keys insted provide your own schema:
    .schema(schema) \
    .load()
 
2) bring it to default 1000 or lower
 
3) are you using new connector? some option seems avialble only for legacy connector
 
Br

@DoredlaCharan