Error MISSING_EXPECTED_PROPERTY during migration to default publishing mode

[minhhung0507]

We are in the process of migrating a pipeline from legacy publishing mode (LIVE virtual schema) to default publishing mode, following the steps outlined in the documentation:
Enable the default publishing mode in a pipeline (Public Preview)

However, when we perform the migration, we encounter the following error:

MISSING_EXPECTED_PROPERTY  
Pipeline with id <pipelineId> failed architecture migration due to missing property <propertyName> on table with id <tableId>. If you wish to proceed with the migration, remove the conf 'pipelines.enableDPMForExistingPipeline' and re-run an update using 'pipelines.setMigrationHints'. After running an update with 'pipelines.setMigrationHints', you can re-enable the conf 'pipelines.enableDPMForExistingPipeline'.

We followed the documented procedure carefully:

1. Paused the pipeline and ensured a recent update had run.

2. remove the conf 'pipelines.enableDPMForExistingPipeline' and re-run an update using 'pipelines.setMigrationHints'

3. After running an update with 'pipelines.setMigrationHints', we re-enable the conf 'pipelines.enableDPMForExistingPipeline'

Despite this, the migration fails with the error above.

Our questions:

• What exactly is the missing property that the error is referring to?

• How can we fix it?

Any advice or troubleshooting steps would be greatly appreciated.

[szymon_dybczak]

Hi @minhhung0507 ,

According to troubleshooting guide the "MISSING_EXPECTED_PROPERTY" error indicates that you did not run a recent update prior to adding the pipelines.enableDPMForExistingPipeline configuration. 

Are you sure that your update was successful?

Enable the default publishing mode in a pipeline | Databricks Documentation

[minhhung0507]

Yes, we did refer to the documentation and followed the recommended troubleshooting steps exactly as described:

We ran a successful update after setting only pipelines.setMigrationHints = true, and then continued from step 3 by re-adding pipelines.enableDPMForExistingPipeline. However, once we attempt another update, the same MISSING_EXPECTED_PROPERTY error appears again.

[mark_ott]

I have 4 steps in my resolution. 

Resolution Steps

1. Remove the configuration pipelines.enableDPMForExistingPipeline from the pipeline settings.

2.  Add (or make sure you have) the configuration pipelines.setMigrationHints set to true.

3. Run an update on the pipeline with these settings. This update fills in necessary metadata/properties on pipeline tables.

4. Once the update completes successfully, re-enable pipelines.enableDPMForExistingPipeline if needed and proceed with the migration