pesky_chris
New Contributor III

Hey,

To clarify, as I think I'm potentially hitting Databricks unintended "functionality".

  • Materialised Views are managed by DLT pipeline, which was deployed with DABs off CI/CD pipeline,
  • DLT Pipeline runs a notebook with Python code creating MVs dynamically,
  • DLT Pipeline is set up with (self-hosted, or non-serverless if you prefer that name) compute,
  • The Compute (effectively VM) as it is span up in the Private Network is set up to have access to other VNets where Postgresql Database is hosted,
  • SQL Warehouse Serverless network configuration is in place (NCC + Private Links to Metastore Storage Account)

What works:

  • I can run DML against those MVs (with non-serverless) Compute,
  • Serverless (SQL Warehouse Serverless) errors but works with everything else

I reckon it is unexpected behaviour. I got admin permissions on the workspace to look up __databricks_internal catalog where delta tables supporting MVs are visible. I traversed DLT's compute spool log to see how they are constructed, the data is streamed into Delta Tables and then reconciled. Furthermore, I can run DML against those technical tables in __databricks_internal (respective schemas) with Serverless without problems.

This is a fragment from the Docs :

> The underlying files supporting materialized views might include data from upstream tables (including possible personally identifiable information) that do not appear in the materialized view definition. This data is automatically added to the underlying storage to support incremental refreshing of materialized views.[...]