How to call a table created with create_table using dlt in a separate notebook?

[rt-slowth]

I created a separate pipeline notebook to generate the table via DLT, and a separate notebook to write the entire output to redshift at the end. The table created via DLT is called spark.read.table("{schema}.{table}").
This way, I can import

[MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE] The materialized view operation query is not allowed: Cannot query the Materialized View main.voyager.shop_silver from an Assigned or No Isolation Shared cluster, please use a Shared cluster or a Databricks SQL warehouse instead.

I get this error. How can I fix it?

[Kaniz · Accepted solution]

Hi @rt-slowth, The error message you received indicates that materialized views in Delta Lake require access to metadata unavailable in a non-shared Databricks job cluster.

Specifically, materialized views rely on access to the Databricks metadata store to perform query op....

Here are some steps to address this issue:

Enable Materialized Views:

• Ensure that materialized view features are enabled for your workspace. If not, reach out to Databricks support to help materialized views for your workspace.
• Consider using DBSQL Serverless (recommended) or Pro warehouse for materialized views.

Cluster Type:

• Avoid using an Assigned or No Isolation Shared cluster for querying materialized views. Instead, use a Shared cluster or a Databricks SQL warehouse.

Review Materialized View Creation:

• Verify that the materialized view creation process adheres to the correct syntax and requirements.
• Ensure the materialized view is correctly created based on your business logic and data requirements.

Check Permissions:

• The user creating the materialized view should have the necessary permissions:
  • SELECT privilege over the base tables referenced by the materialized view.
  • USAGE privilege on the catalog and schema containing the materialized view.

Refresh Schedule:

• If you’re scheduling refreshes for materialized views, ensure the cron schedule is correctly configured.
• Note that SCHEDULE REFRESH cannot be used in a Delta Live Tables pipeline definition.

[rt-slowth]

Enable Materialized Views:

- Consider using DBSQL Serverless (recommended) or Pro warehouse for materialized views.

- Ensure that materialized view features are enabled for your workspace.

Can you point me to the documentation for this workaround?

[rt-slowth]

@Kaniz 

Enable Materialized Views:

• Ensure that materialized view features are enabled for your workspace.
• Consider using DBSQL Serverless (recommended) or Pro warehouse for materialized views.

Can you point me to the documentation for this workaround?

[Kaniz · Accepted solution]

Here it is - https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/materialized-views/materializ...

[Kaniz]

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!