How to set pipelines.incompatibleViewCheck.enabled = false

159312
New Contributor III

‎08-16-2022 12:36 PM

I tried to load a static table as source to a streaming dlt pipeline. I understand this is not optimum, but it provides the best path toward eventually having a full streaming pipeline. When I do I get the following error:

pyspark.sql.utils.AnalysisException: View 'raw_public_streamme' is not a streaming view and must be referenced using read. This check can be disabled by setting Spark conf pipelines.incompatibleViewCheck.enabled = false.

How can I set pipelines.incompatibleViewCheck.enabled = false? Doing so with spark.conf.set does not seem to have any effect.

Thanks.

Labels:
0 Kudos

kfoster
Databricks Partner

‎08-30-2022 04:57 AM

when you declare a table or view, you can pass use something as this:

 @dlt.table(   
  spark_conf={
   "pipelines.incompatibleViewCheck.enabled": "false"
  }
)

0 Kudos