Unable to communicate with AWS DocumentDB
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
Runtime version:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
The error seems to be related to writing data to a MongoDB data source, as indicated by the com.mongodb.spark.sql.connector.exceptions.DataException
.
It appears that the error is occurring during the execution of a Spark job that involves writing data to a MongoDB data source. The error message shows that the write operation was aborted for a specific partition and task, and manual data cleanup may be required.
Here are some steps you can take to troubleshoot and resolve this issue:
-
Check MongoDB Connection and Configuration: Ensure that the MongoDB connection details and configurations are correct. Verify that the MongoDB server is running and accessible from the Spark cluster.
-
Review Data Schema and Types: Ensure that the data being written to MongoDB matches the expected schema and data types. Any discrepancies in the schema or data types can cause write failures.
-
Check for Data Skew: Data skew can cause certain partitions to have significantly more data than others, leading to task failures. Review the data distribution and consider repartitioning the data to balance the load.
-
Increase Resources: If the task is failing due to resource constraints, consider increasing the resources allocated to the Spark job, such as executor memory and cores.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
Im not sure that is the problem as stated by the error from the job stage:
'''
com.mongodb.MongoTimeoutException: Timed out while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=UNKNOWN, servers=[{address=<<REDACTED>>:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketWriteException: Exception sending message}, caused by {javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}}]
'''
Id also like to point out that I did verify connection from inside the node:
'''
%sh openssl s_client -connect <REDACTED>:27017 -CAfile /Workspace/Users/<REDACTED>/rds-combined-ca-bundle.pem
'''