Databricks Community

bmhardy

I am looking to build a hierarchy from a parent child relationship table, which I would typically use a recursive statement for in SQL Server / Azure SQL. This would mean setting an anchor, most commonly the top record of the tree, and then join back...

bmhardy · 11-12-2024

I have just learnt that recursive CTE's are not supported in Databricks SQL, however, we are looking to shift the complex aggregations into Databricks instead of relying on Azure SQL DB.We are using Azure SQL DB with CDC enabled in combination with A...

bmhardy

Thank you, I will give this a try. I'll let you know how it goes.

bmhardy · 12-11-2024

Is this post referring to Direct Publishing Mode? As we are multi-tenanted we have to have separate schema per client, which currently means a single pipeline per client. This is not cost effective at all, so we are very much reliant on DPM. I believ...

bmhardy · 11-19-2024

We have gone down a different route where we are using SQL for our calculated layer and then a Python notebook for our aggregated layer. It is much easier to roll up data using a Python UDF than it is trying to work out how to do it in SQL.

bmhardy · 11-13-2024

Thank you Amit.We use a 5 layer process, instead of the traditional medallion bronze, silver, gold. The materialised views are split into two functions instead of just gold, where we have 'calculated' and 'aggregated'. Once we have performed any calc...

Databricks Community

User Stats

User Activity

Creating a hierarchy without recursive statements

Hierarchy roll up aggregation

Re: Creating a hierarchy without recursive statements

Re: Delta Live Table name dynamically

Re: Hierarchy roll up aggregation

Re: Hierarchy roll up aggregation