cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks-PowerBI-Architecture-Help

venkat-bodempud
New Contributor III

Hello Community,

We are currently working designing Power BI reports, the data source is databricks. We have all our reporting data in bronze/silver layer of databricks. we want to create summarized/aggregated tables in Gold layer and we want to connect to gold layer tables from Power BI.

Currently we are looking for solution design help in moving data from bronze/silver layer to gold layer as efficient as possible. This is going to near real time solution. we have schedule the pipelines as well for incremental loading.

please share your suggestions and let us know incase of questions.

Thanks in advance.

Thanks

Venkat

3 REPLIES 3

hitech88
New Contributor II

Hello,

I assume you would be using spark to read data from bronze/silver layer to transform and calculate the summarized/aggregated tables and load it into gold layer.

Additionally, using delta tables especially in gold layer for incremental upserts will allow for efficient processing and updating of data in gold layer tables.

For consumption of data via Power BI, you can also try Databricks SQL which allows for fast response time on large scale data in data lake storage, by leveraging features like delta cache and photon under the hood and is already optimized so no manual configurations or tweaking required.

venkat-bodempud
New Contributor III

Hi @Hitesh Sahni​ , thanks for your reply.

Anonymous
Not applicable

Hi @bodempudi venkat​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group