topic Re: Databricks-PowerBI-Architecture-Help in Data Engineering

Databricks-PowerBI-Architecture-Help

venkat-bodempud — Fri, 03 Feb 2023 10:14:11 GMT

Hello Community,

We are currently working designing Power BI reports, the data source is databricks. We have all our reporting data in bronze/silver layer of databricks. we want to create summarized/aggregated tables in Gold layer and we want to connect to gold layer tables from Power BI.

Currently we are looking for solution design help in moving data from bronze/silver layer to gold layer as efficient as possible. This is going to near real time solution. we have schedule the pipelines as well for incremental loading.

please share your suggestions and let us know incase of questions.

Thanks in advance.

Thanks

Venkat

Re: Databricks-PowerBI-Architecture-Help

hitech88 — Sat, 04 Feb 2023 20:05:48 GMT

Hello,

I assume you would be using spark to read data from bronze/silver layer to transform and calculate the summarized/aggregated tables and load it into gold layer.

Additionally, using delta tables especially in gold layer for incremental upserts will allow for efficient processing and updating of data in gold layer tables.

For consumption of data via Power BI, you can also try Databricks SQL which allows for fast response time on large scale data in data lake storage, by leveraging features like delta cache and photon under the hood and is already optimized so no manual configurations or tweaking required.

Re: Databricks-PowerBI-Architecture-Help

venkat-bodempud — Thu, 16 Feb 2023 09:40:02 GMT

Hi @Hitesh Sahni , thanks for your reply.

Re: Databricks-PowerBI-Architecture-Help

Anonymous — Sun, 09 Apr 2023 07:01:23 GMT

Hi @bodempudi venkat

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!