Performance issue when reading data from a view.

sensanjoy
Contributor II

Hi All,

We are facing some performance issue and I need your help to know what could be the best approach to follow here.

Existing:

  1.  For each region, we have view(Reg1_View,Reg2_View..) to pull data from table( we don't have direct access to table).
  2. And then we are combining data from all view into another view named All_Reg_View(Reg1_View union all Reg2_View..).
  3. Using the combining view All_Reg_View in join condition with other view in notebook and create target delta table.

View_Vs_Staging_Table.png

All views are constructed same way and we are dealing with billions of data.

Proposed approach:

  1. Create one staging table(truncate & load) replacing the combined view(All_Reg_View) by performing union of all views.
  2. Partition the stg table based on region.
  3. Use the stg tables in joining to create target delta table.

Please let me know your thought.