aleksandra_ch
Databricks Employee
Databricks Employee

Hi @FAHADURREHMAN ,

In order to improve performance of the Materialized View:

  • First, land parquet files into a Streaming Table. Use Auto Loader for incremental ingestion:
    CREATE OR REFRESH STREAMING TABLE raw AS
    SELECT * FROM STREAM read_files(s3_path, format => 'parquet');​
  • Use this Streaming Table as a source of the Materialized View.
  • The Materialized View will get incrementally refreshed, meaning that the joins and aggregations will be recalculated for the newly ingested rows only. 
    Prerequisites for incremental refresh:

With this setup, you make sure the pipeline runs as fast as possible by minimal recomputations.

Hope it helps.

Best regards,