Databricks Destiny with Fivetrans best practices

Zoraida
New Contributor

‎09-25-2024 05:52 AM

Hello! we are trying to use Fivetran for ingesting different sources into the data lake so we will have multiple connectors. We would like to know what are the recommendations when selecting the SQL warehouses. Since the new serverless SQL warehouses are capable of autoscaling, we were wondering if having a single Destiny(since the idea is to have a catalog by environment so we will have multiple schemas into the same catalog but coming from different sources) would be enough. We would like to understand pros/cons for not only performance but also cost.

Thanks!

0 Kudos

SathyaSDE
Contributor

‎10-25-2024 11:53 PM - edited ‎10-25-2024 11:56 PM

Hi,

To understand about the Databricks SQL Serverless cost, you can see here - https://www.databricks.com/product/pricing/databricks-sql

In terms of comparison, Databricks is said to be the most cost efficient & high performant in the market amongst its competitors who provides Serverless offerings.

There is also new feature introduced to manage the cost associated with Serverless usage - Serverless budget policy

However, as you had asked the cons, here are some limitations with Serverless compute - https://docs.databricks.com/en/compute/serverless/limitations.html

In addition to this - Serverless compute plane networking products might incur data transfer costs

I hope this helps!!

0 Kudos