topic Re: Has the numUpdateRetryAttempts property been deprecated? in Data Engineering

Has the numUpdateRetryAttempts property been deprecated?

guangyi — Wed, 23 Oct 2024 04:29:45 GMT

I noticed there is a numUpdateRetryAttempts property mentioned in the the document https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/properties used for configing the retry times of any DLT pipeline, but I cannot find it in the DLT pipeline creation API document https://docs.databricks.com/api/workspace/pipelines/create.

Also if I added it into the asset yaml bundle file like

silver_pipeline: name: clusters: photon: pipelines.numUpdateRetryAttempts: 3

The validator gave me a warning about it is a unknown field

So has the numUpdateRetryAttempts property been deprecated?

Re: Has the numUpdateRetryAttempts property been deprecated?

VZLA — Thu, 31 Oct 2024 16:19:17 GMT

According to the Delta Live Tables properties reference, pipelines.numUpdateRetryAttempts is a recognized configuration parameter. It specifies the maximum number of attempts to retry an update before failing the update when a retryable failure occurs. This parameter is set as a key-value pair in the pipeline configuration and is applicable to triggered updates run in production mode. The default value for this property is five.

{ "configuration": { "pipelines.numUpdateRetryAttempts": 5 } }

Or:

silver_pipeline: name: <your_pipeline_name> clusters: - <cluster_configuration> photon: <true_or_false> configuration: pipelines.numUpdateRetryAttempts: 3

Hope it helps!