3 weeks ago
Hi All,
We are facing some performance issue and I need your help to know what could be the best approach to follow here.
Existing:
All views are constructed same way and we are dealing with billions of data.
Proposed approach:
Please let me know your thought.
3 weeks ago
Hi @sensanjoy,
Could you please advise what kind of performance issue are you facing? Is it slowness while querying the views?
3 weeks ago
@Alberto_Umana Yes, during query execution and joining back with other views resulting more time. We are building target table by performing joining between views created like this way(combining multiple region views).
3 weeks ago
To address the performance challenges associated with All_Reg_View, I propose replacing it with a partitioned staging table. This change will significantly enhance query efficiency, simplify execution plans, and prepare the pipeline for future scalability.
Create a Partitioned Staging Table
Replace All_Reg_View in Joins
This solution ensures more robust and efficient data processing while supporting the pipeline's long-term goals.
2 weeks ago
Thanks @fmadeiro . We are doing union all between all region like Reg1_View union all Reg2_View..Regn_View to get the combined dataset into stg table. Is there any other efficient way to do it!!
2 weeks ago
Does any table hold data of all region
1. if yes. Get a Materialized view created (replacing all_reg_view)
2. i see you already tried creating a staging table replacing the all_reg_view.
Try creating cluster key along with partition.
Cluster key on the region (joining columns) and partition on some date column if available.
3. Analyze the other views for performance which you are joining with All_reg_view.
2 weeks ago
We dont have access to the table level directly . Its only region level views are accessible and we union them to get combine data.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group