Error MISSING_EXPECTED_PROPERTY during migration to default publishing mode

minhhung0507
Valued Contributor

‎08-04-2025 09:35 PM - edited ‎08-04-2025 09:37 PM

We are in the process of migrating a pipeline from legacy publishing mode (LIVE virtual schema) to default publishing mode, following the steps outlined in the documentation:
Enable the default publishing mode in a pipeline (Public Preview)

However, when we perform the migration, we encounter the following error:

MISSING_EXPECTED_PROPERTY  
Pipeline with id <pipelineId> failed architecture migration due to missing property <propertyName> on table with id <tableId>. If you wish to proceed with the migration, remove the conf 'pipelines.enableDPMForExistingPipeline' and re-run an update using 'pipelines.setMigrationHints'. After running an update with 'pipelines.setMigrationHints', you can re-enable the conf 'pipelines.enableDPMForExistingPipeline'.

We followed the documented procedure carefully:

  1. Paused the pipeline and ensured a recent update had run.

  2. remove the conf 'pipelines.enableDPMForExistingPipeline' and re-run an update using 'pipelines.setMigrationHints'

  3. After running an update with 'pipelines.setMigrationHints', we re-enable the conf 'pipelines.enableDPMForExistingPipeline'

Despite this, the migration fails with the error above.

Our questions:

  • What exactly is the missing property that the error is referring to?

  • How can we fix it?

Any advice or troubleshooting steps would be greatly appreciated.

Regards,
Hung Nguyen
0 Kudos

szymon_dybczak
Esteemed Contributor III

‎08-05-2025 12:05 AM

Hi @minhhung0507 ,

According to troubleshooting guide the "MISSING_EXPECTED_PROPERTY" error indicates that you did not run a recent update prior to adding the pipelines.enableDPMForExistingPipeline configuration. 

Are you sure that your update was successful?

Enable the default publishing mode in a pipeline | Databricks Documentation

0 Kudos

minhhung0507
Valued Contributor
In response to szymon_dybczak

‎08-05-2025 12:09 AM

Yes, we did refer to the documentation and followed the recommended troubleshooting steps exactly as described:

We ran a successful update after setting only pipelines.setMigrationHints = true, and then continued from step 3 by re-adding pipelines.enableDPMForExistingPipeline. However, once we attempt another update, the same MISSING_EXPECTED_PROPERTY error appears again.

Regards,
Hung Nguyen
0 Kudos

mark_ott
Databricks Employee
Databricks Employee
In response to minhhung0507

‎09-22-2025 11:46 AM

I have 4 steps in my resolution. 

Resolution Steps

1. Remove the configuration pipelines.enableDPMForExistingPipeline from the pipeline settings.

2.  Add (or make sure you have) the configuration pipelines.setMigrationHints set to true.

3. Run an update on the pipeline with these settings. This update fills in necessary metadata/properties on pipeline tables.

4. Once the update completes successfully, re-enable pipelines.enableDPMForExistingPipeline if needed and proceed with the migration