cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to set pipelines.incompatibleViewCheck.enabled = false

159312
New Contributor III

I tried to load a static table as source to a streaming dlt pipeline. I understand this is not optimum, but it provides the best path toward eventually having a full streaming pipeline. When I do I get the following error:

pyspark.sql.utils.AnalysisException: View 'raw_public_streamme' is not a streaming view and must be referenced using read. This check can be disabled by setting Spark conf pipelines.incompatibleViewCheck.enabled = false.

How can I set pipelines.incompatibleViewCheck.enabled = false? Doing so with spark.conf.set does not seem to have any effect.

Thanks.

1 REPLY 1

kfoster
Contributor

when you declare a table or view, you can pass use something as this:

 @dlt.table(   
  spark_conf={
   "pipelines.incompatibleViewCheck.enabled": "false"
  }
)

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now