topic Re: Need help to insert huge data into cosmos db from azure data lake storage using databricks in Data Engineering

Need help to insert huge data into cosmos db from azure data lake storage using databricks

manasa — Fri, 15 Jul 2022 08:30:11 GMT

I am trying to insert 6GB of data into cosmos db using OLTP Connector

Container RU's:40000

Cluster Config:

cfg = { 
  "spark.cosmos.accountEndpoint" : cosmosdbendpoint,
  "spark.cosmos.accountKey" : cosmosdbmasterkey,
  "spark.cosmos.database" : cosmosdatabase,
  "spark.cosmos.container" : cosmosdbcontainer,
}
 
spark.conf.set("spark.sql.catalog.cosmosCatalog", "com.azure.cosmos.spark.CosmosCatalog")
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountEndpoint", cosmosdbendpoint)
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountKey", cosmosdbmasterkey)
spark.conf.set("spark.cosmos.write.bulk.enabled", "true")
 
json_df.write.format("cosmos.oltp").options(**cfg).mode("APPEND").save()

It is taking around 3hrs for me to load into cosmos db

1.Is increasing RU's is the only approach to decrease the execution time

2.Other than OLTP connector, do we have any ways to insert bulk data within less time

3.How to calculate RU's based on data size

Re: Need help to insert huge data into cosmos db from azure data lake storage using databricks

manasa — Fri, 15 Jul 2022 12:45:11 GMT

Hi @Kaniz Fatma , my problem is not with the resources. I tried every thing mentioned in the article but I need to insert bulk data in less time(def not 3hrs for 6gb data).So, I am looking for a optimized way.

Re: Need help to insert huge data into cosmos db from azure data lake storage using databricks

SteveMeckstroth — Sun, 12 Feb 2023 03:06:03 GMT

You have probably found a solution, but for others that end up here I got dramatic improvements using the Mongo connector to CosmosDB: https://www.mongodb.com/docs/spark-connector/current/write-to-mongodb/

Re: Need help to insert huge data into cosmos db from azure data lake storage using databricks

ImAbhishekTomar — Wed, 22 Feb 2023 21:31:08 GMT

Did anyone find solution for this, I’m also using similar clutter and RAU and data ingestion taking lot of time….?