topic DATABRICKS TO AZ SQL?? in Data Engineering

DATABRICKS TO AZ SQL??

Ruby8376 — Tue, 17 Oct 2023 15:26:12 GMT

Hi All,,

quick question:

Is this correct data flow pattern: Databricks -> Az SQL -> Tableau??

Or does it have to go through ADLS: Databricks -> ADLS -> Az SQL - > Tableau?

Also, is it better to leverage databricks lakehouse sql warehouse capability as it has native tableau connector?

@Retired_mod @-werners-

Thanks!

Re: DATABRICKS TO AZ SQL??

-werners- — Tue, 17 Oct 2023 15:26:24 GMT

Both are possible,
but I prefer saving data on a data lake first before putting it in a database (option 2).

Re: DATABRICKS TO AZ SQL??

Ruby8376 — Tue, 17 Oct 2023 15:30:39 GMT

Also, is it better to leverage databricks lakehouse sql warehouse capability as it has native tableau connector?

Re: DATABRICKS TO AZ SQL??

-werners- — Wed, 18 Oct 2023 05:23:35 GMT

I would not call it 'better' per se. A lakehouse is a more modern approach to a classic datawarehouse, using flexible distributed cloud compute, cheap storage and open file formats.
If you have an existing environment, which works well, that is heavily tied to a certain vendor (like MS f.e.) a lakehouse might not be for you (although it is perfectly possible to do so).
But if you need to reengineer an existing warehouse, or start over,... I would definitely consider a lakehouse.