pipeline config DAB

[prakharsachan]

I am deploying DLT pipeline in dev environment using DABs. source code is in a python script file. In the pipeline's yml file the configuration key is set to true(with all correct indentations), yet the pipeline isnt deploying in the continuous mode. Whats the reason?

[szymon_dybczak]

Hi @prakharsachan ,

Continuous must be set inside the pipeline resource definition, not under configuration.

The configuration block in a SDP (former DLT) pipeline definition is for Spark/pipeline settings (key-value string pairs passed to the runtime), not for pipeline deployment properties. So if you have something like:

Databricks will just ignore it as an unrecognized Spark config. The fix is to promote continuous to a top-level property of the pipeline resource:

[prakharsachan · Accepted solution]

Hi @szymon_dybczak , just to clarify: I am already using continuous as a top-level property. However, I recently found a community post(replied by databricks employee) explaining that when a deployment occurs in development mode, the continuous setting is bypassed and the pipeline won't deploy in that mode—even if the property is set to true. This setting only takes effect when deploying in production mode. I have also tested this fact.