how to connect to aws elasticsearch of another account from databricks

[niladri]

Hi -  I have tried my level best to go through both elasticsearch documentation as well as Databricks documentation to get an answer for my question - is it possible to connect to AWS elasticsearch of a different AWS account from Databricks? I did not find an answer. I am writing code in PySpark.

I installed org.elasticsearch:elasticsearch-spark-30_2.12:7.17.6 

Apache Spark - 3.12 , DBR : Standard 9.2 LTS

as mentioned in Databricks documentation and was running the following code. It did not help at all.

df = (spark.read

.format( "org.elasticsearch.spark.sql" )

.option( "es.nodes", hostname )

.option( "es.port", port )

.option( "es.net.ssl", 'true' )

.option( "es.nodes.wan.only", "true" )

.load( f"{index}" )

)

The error I am getting is : AccessControlException: Permission denied: user [***] does not have [read] privilege on [dbfs:/***] ( I have masked the user and the ES name ) 

I think the right way is to do an assume role using boto3 and then somehow configure Spark session to assume the IAM role. Am I correct?

Is it not do-able through PySpark?

[Kaniz]

Hi @niladri , It's possible to connect to AWS Elasticsearch of a different AWS account from Databricks.
- The error is related to permissions, indicating the user or role lacks necessary access permissions.
- To solve this, use AWS SDK boto3 to assume a role with necessary permissions.