Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-19-2025 03:23 AM
Probably, this syntax is more correct:
def save_to_open_search(df):
writer = (
df.write.format("opensearch")
.option("opensearch.nodes", host)
.option("opensearch.port", port)
.option("pushdown", "true")
.option("opensearch.batch.write.refresh", "false")
.option("opensearch.mapping.id", "id")
.option("opensearch.write.operation", "upsert")
.option("opensearch.aws.sigv4.enabled", "true")
.option("opensearch.aws.sigv4.region", "us-west-2")
.option("opensearch.nodes.resolve.hostname", "false")
.option("opensearch.nodes.wan.only", "true")
.option("opensearch.net.ssl", "true")
.option("opensearch.batch.size.entries", "5000")
.option("opensearch.batch.size.bytes", "10mb")
)
writer.mode("append").save(index)But I still have the same error:
Py4JJavaError: An error occurred while calling o615.save.
: org.opensearch.hadoop.OpenSearchHadoopIllegalArgumentException: Cannot detect OpenSearch version - typically this happens if the network/OpenSearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'opensearch.nodes.wan.only'...