cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to set pipelines.incompatibleViewCheck.enabled = false

159312
New Contributor III

I tried to load a static table as source to a streaming dlt pipeline. I understand this is not optimum, but it provides the best path toward eventually having a full streaming pipeline. When I do I get the following error:

pyspark.sql.utils.AnalysisException: View 'raw_public_streamme' is not a streaming view and must be referenced using read. This check can be disabled by setting Spark conf pipelines.incompatibleViewCheck.enabled = false.

How can I set pipelines.incompatibleViewCheck.enabled = false? Doing so with spark.conf.set does not seem to have any effect.

Thanks.

1 REPLY 1

kfoster
Contributor

when you declare a table or view, you can pass use something as this:

 @dlt.table(   
  spark_conf={
   "pipelines.incompatibleViewCheck.enabled": "false"
  }
)

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.