topic Re: How to call a table created with create_table using dlt in a separate notebook? in Data Engineering

How to call a table created with create_table using dlt in a separate notebook?

rt-slowth — Thu, 30 Nov 2023 23:28:57 GMT

I created a separate pipeline notebook to generate the table via DLT, and a separate notebook to write the entire output to redshift at the end. The table created via DLT is called spark.read.table("{schema}.{table}").
This way, I can import

[MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE] The materialized view operation query is not allowed: Cannot query the Materialized View main.voyager.shop_silver from an Assigned or No Isolation Shared cluster, please use a Shared cluster or a Databricks SQL warehouse instead.

I get this error. How can I fix it?

Re: How to call a table created with create_table using dlt in a separate notebook?

rt-slowth — Mon, 04 Dec 2023 05:34:11 GMT

Enable Materialized Views:

- Consider using DBSQL Serverless (recommended) or Pro warehouse for materialized views.

- Ensure that materialized view features are enabled for your workspace.

Can you point me to the documentation for this workaround?

Re: How to call a table created with create_table using dlt in a separate notebook?

rt-slowth — Mon, 04 Dec 2023 05:40:10 GMT

@Retired_mod

Enable Materialized Views:

Ensure that materialized view features are enabled for your workspace.
Consider using DBSQL Serverless (recommended) or Pro warehouse for materialized views.

Can you point me to the documentation for this workaround?