I tried to load a static table as source to a streaming dlt pipeline. I understand this is not optimum, but it provides the best path toward eventually having a full streaming pipeline. When I do I get the following error:
pyspark.sql.utils.AnalysisException: View 'raw_public_streamme' is not a streaming view and must be referenced using read. This check can be disabled by setting Spark conf pipelines.incompatibleViewCheck.enabled = false.
How can I set pipelines.incompatibleViewCheck.enabled = false? Doing so with spark.conf.set does not seem to have any effect.
Thanks.