topic Re: Materialized Views Without DLT? in Get Started Discussions

Materialized Views Without DLT?

ChristianRRL — Tue, 04 Jun 2024 14:55:33 GMT

I'm curious, is DLT *required* to use Materialized Views in Databricks? Is it not possible to create and refresh a Materialized view via a standard Databricks Workflow?

Re: Materialized Views Without DLT?

raphaelblg — Tue, 04 Jun 2024 15:30:16 GMT

Hi @ChristianRRL,

There are DLT materialized views: https://docs.databricks.com/en/delta-live-tables/index.html#materialized-view

and

Databricks SQL Materialized Views: https://docs.databricks.com/en/sql/user/materialized-views.html#use-materialized-views-in-databricks-sql.

Re: Materialized Views Without DLT?

ChristianRRL — Tue, 04 Jun 2024 20:03:30 GMT

Hi @raphaelblg , sorry but I think you misunderstood my question. I'm aware that I can create a DLT pipeline from scratch to create Materialized Views, but I was surprised when I was attempting to create a Materialized View without trying to use DLT, but when I ran this in a standard notebook (connected to our configured cluster) I see that it does seem require DLT:

My question is, either via a Python or SQL notebook cell, can I (1) create, and (2) refresh a Materialized View without requiring DLT at all???

Re: Materialized Views Without DLT?

ChristianRRL — Tue, 04 Jun 2024 20:07:14 GMT

For additional clarity, I would like to show that when I attempt to run this in the SQL Editor with the SQL Starter Warehouse, I get an error because we do not have UC enabled. But I'm wanting to get the Materialized View functionality without fully requiring DLT or UC:

Re: Materialized Views Without DLT?

raphaelblg — Tue, 04 Jun 2024 20:50:44 GMT

@ChristianRRL, You either need a DLT pipeline or a SQL Warehouse with UC access.

"In Databricks SQL, materialized views are Unity Catalog managed tables that allow users to precompute results based on the latest version of data in source tables. ".

At: https://docs.databricks.com/en/sql/user/materialized-views.html#what-are-materialized-views

Re: Materialized Views Without DLT?

gbattawk — Thu, 16 Jan 2025 14:13:54 GMT

Hi @raphaelblg ,
I am also not able to create mview through UC enabled SQL Warehouse (Pro). I get error: Failed to start the DLT service on cluster xxx-xxxx-xxxxxx-xxx. An Azure storage request was not authorized. The storage account's 'Firewalls and virtual networks' settings may be blocking access to storage services. Validate your Azure storage credentials or firewall exception settings. Why does the create MV functionality need serverless compute if I want to stay with Pro compute.

Re: Materialized Views Without DLT?

hari-prasad — Thu, 16 Jan 2025 14:43:12 GMT

Hi @ChristianRRL ,

When creating a materialized view in Databricks, the data is stored in DBFS, cloud storage, or Unity Catalog volume. You can still create a materialized view by overwriting the same table each time, instead of using Append, Update, or Delete operations. Be sure to manage the delta table retention policy and perform vacuuming for optimal performance.

Regards,
Hari Prasad