topic Re: Delta sharing vs CosmosDB in Get Started Discussions

Delta sharing vs CosmosDB

Phani1 — Wed, 18 Dec 2024 07:09:42 GMT

Hi All,

We have a situation where we write data to CosmosDB and create JSON data for a transaction table, which includes a mini statement in JSON format.

Now, we want to introduce the concept of delta sharing and share the transaction table. The Java application will access the delta table to generate JSON data, and there could be hundreds of API calls to the delta sharing table.

We would like to understand the costs involved in this scenario when comparing delta sharing(computation costs, egress cost etc ) to CosmosDB. Additionally, we want to know if there are any performance issues when accessing the delta sharing table concurrently through API calls to generate JSON data.

Regards,

Phani

Re: Delta sharing vs CosmosDB

Walter_C — Wed, 18 Dec 2024 11:39:50 GMT

Costs:

Egress Costs:
- Delta Sharing: Within the same region, Delta Sharing incurs no egress costs. However, sharing data across different regions or clouds may result in egress fees charged by the cloud provider. Databricks supports sharing from Cloudflare R2, which incurs no egress fees. Tools and recommendations are available to monitor and avoid egress fees (source: "Monitor and manage Delta Sharing egress costs (for providers)").
- CosmosDB: Egress costs apply when data is read from CosmosDB and transferred out of the Azure region. The exact costs depend on the amount of data transferred and the regions involved.
Computation Costs:
- Delta Sharing: The data provider incurs storage costs for Delta or Parquet files, but there are no compute costs on the provider's side for sharing data. The recipient incurs data processing costs when they access and process the shared data (source: "Delta Sharing Cost Breakdown").
- CosmosDB: Costs are associated with the provisioned throughput (RU/s) for read and write operations, as well as storage costs.

Performance Issues:

Delta Sharing: There can be performance issues when accessing the Delta Sharing table concurrently through API calls. For example, if the number of API requests exceeds the predefined rate limit, it can result in "Too Many Requests" errors (HTTP 429).
CosmosDB: CosmosDB is designed to handle high throughput and low latency for concurrent operations. However, performance can be impacted if the provisioned throughput is not sufficient for the workload

Re: Delta sharing vs CosmosDB

Phani1 — Fri, 20 Dec 2024 09:24:48 GMT

Thanks for your reply,

Right now, the team is transferring data from Databricks to Cosmos DB, and then they're using REST APIs to access that data. They handle about 100 requests per minute, with some tables needing around 100 requests per second due to high transactions.

However, the customer wants to move away from Cosmos DB because it requires a lot of effort to write the data,
and it's becoming too expensive. They are looking for alternative ways to access the delta table data using REST APIs through Java. Can you please provide any approach on this.

Re: Delta sharing vs CosmosDB

Walter_C — Fri, 20 Dec 2024 12:14:06 GMT

You could use a JDBC connection to connect to a cluster or SQL warehouse and from there you could run your SQL commands to query the delta tables https://docs.databricks.com/en/integrations/jdbc/index.html