Databricks Destiny with Fivetrans best practices

Zoraida — Wed, 25 Sep 2024 12:52:30 GMT

Hello! we are trying to use Fivetran for ingesting different sources into the data lake so we will have multiple connectors. We would like to know what are the recommendations when selecting the SQL warehouses. Since the new serverless SQL warehouses are capable of autoscaling, we were wondering if having a single Destiny(since the idea is to have a catalog by environment so we will have multiple schemas into the same catalog but coming from different sources) would be enough. We would like to understand pros/cons for not only performance but also cost.

Thanks!

Re: Databricks Destiny with Fivetrans best practices

SathyaSDE — Sat, 26 Oct 2024 06:56:02 GMT

Hi,

To understand about the Databricks SQL Serverless cost, you can see here - https://www.databricks.com/product/pricing/databricks-sql

In terms of comparison, Databricks is said to be the most cost efficient & high performant in the market amongst its competitors who provides Serverless offerings.

There is also new feature introduced to manage the cost associated with Serverless usage - Serverless budget policy

However, as you had asked the cons, here are some limitations with Serverless compute - https://docs.databricks.com/en/compute/serverless/limitations.html

In addition to this - Serverless compute plane networking products might incur data transfer costs

I hope this helps!!

topic Databricks Destiny with Fivetrans best practices in Get Started Discussions

Databricks Destiny with Fivetrans best practices

Re: Databricks Destiny with Fivetrans best practices