SDP: Your table schema requires manually enablement of the following table feature(s): timestampNtz.

liquibricks
Databricks Partner

Trying to create a Spark Declarative Pipeline with a Materialized View based on a Streaming Table. The Streaming Table works fine, but when trying to add a Materialized View the pipeline fails to initialize with:

Your table schema requires manually enablement of the following table feature(s): timestampNtz.

To do this, run the following command for each of features listed above: ALTER TABLE table_name SET TBLPROPERTIES ('delta.feature.feature_name' = 'supported') Replace "table_name" and "feature_name" with real values. Current supported feature(s): appendOnly, changeDataFeed, deletionVectors, invariants.

 

The MV does not exist so I can't set the table properties. I found a thread about setting it in the Spark context, however setting 

"spark.databricks.delta.properties.defaults.feature.timestampNtz" to "enabled" in the Spark context doesn't seem to have any effect. Any ideas as to what's wrong?

 

stbjelcevic
Databricks Employee
Databricks Employee

Hi @liquibricks ,

I see you said that the materialized view does not exist, so you can't run the ALTER TABLE command. Try to add TBLPROPERTIES ('delta.feature.timestampNtz'='supported') to the MV CREATE statement (or pass it via table_properties in your pipeline definition). Don’t rely on Spark defaults for materialized views as they won’t be applied.

View solution in original post

Thanks @stbjelcevic ! Setting it as a table property appears to have worked. My pipeline now has other problems to address so it's not working yet, but that's a different thread 😅

 

            @dp.materialized_view(
               name=f"{mycatalog}.{myschema}.mv_{mytable}",
               table_properties={"delta.feature.timestampNtz": "supported"}
            )
            def create_dynamic_mv():
                ....