I am try to connect to cosmos using spark cosmos connector following this instruction: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/tutorial-spark-connector?pivots=programming-....

The problem is the all spark cosmos operation keep loading and never populate any result. And I could confirm that this problem is not related to incorrect key, since I can use the same key query data from Cosmos using cosmos client python SDK.

Note: All the cloud resource including databricks workspace and cosmos are in private VNET that peers together.

Here is my code:

cfg = {
  "spark.cosmos.accountEndpoint" : cosmosEndpoint,
  "spark.cosmos.accountKey" : cosmosMasterKey,
  "spark.cosmos.database" : cosmosDatabaseName,
  "spark.cosmos.container" : cosmosContainerName
}

df = spark.read.format("cosmos.oltp").options(**cfg)\
 .option("spark.cosmos.read.inferSchema.enabled", "true")\
 .load()

Here is my cluster config:

- Databrick runtime: 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)

- Access mode: [Legacy] Single user standard (I try using other access mode already but not working)

- Installed libs: com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12:4.37.2