how to connect to aws elasticsearch of another account from databricks

niladri — Thu, 28 Sep 2023 06:56:05 GMT

Hi - I have tried my level best to go through both elasticsearch documentation as well as Databricks documentation to get an answer for my question - is it possible to connect to AWS elasticsearch of a different AWS account from Databricks? I did not find an answer. I am writing code in PySpark.

I installed org.elasticsearch:elasticsearch-spark-30_2.12:7.17.6

Apache Spark - 3.12 , DBR : Standard 9.2 LTS

as mentioned in Databricks documentation and was running the following code. It did not help at all.

df = (spark.read

.format( "org.elasticsearch.spark.sql" )

.option( "es.nodes", hostname )

.option( "es.port", port )

.option( "es.net.ssl", 'true' )

.option( "es.nodes.wan.only", "true" )

.load( f"{index}" )

)

The error I am getting is : AccessControlException: Permission denied: user [***] does not have [read] privilege on [dbfs:/***] ( I have masked the user and the ES name )

I think the right way is to do an assume role using boto3 and then somehow configure Spark session to assume the IAM role. Am I correct?

Is it not do-able through PySpark?

topic how to connect to aws elasticsearch of another account from databricks in Data Engineering

how to connect to aws elasticsearch of another account from databricks