I am try to connect to cosmos using spark cosmos connector following this instruction: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/tutorial-spark-connector?pivots=programming-....
The problem is the all spark cosmos operation keep loading and never populate any result. And I could confirm that this problem is not related to incorrect key, since I can use the same key query data from Cosmos using cosmos client python SDK.
Note: All the cloud resource including databricks workspace and cosmos are in private VNET that peers together.
Here is my code:
cfg = {
"spark.cosmos.accountEndpoint" : cosmosEndpoint,
"spark.cosmos.accountKey" : cosmosMasterKey,
"spark.cosmos.database" : cosmosDatabaseName,
"spark.cosmos.container" : cosmosContainerName
}
df = spark.read.format("cosmos.oltp").options(**cfg)\
.option("spark.cosmos.read.inferSchema.enabled", "true")\
.load()
Here is my cluster config:
- Databrick runtime: 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)
- Access mode: [Legacy] Single user standard (I try using other access mode already but not working)
- Installed libs: com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12:4.37.2