Delta Sharing CDF API error: "RESOURCE_LIMIT_EXCEEDED"

[Data_Analytics1]

Hi,

When attempting to read a particular version from the Databricks Delta Sharing CDF (Change Data Feed) API, even when that version contains only one data file, an error occurs due to a timeout with following message:

"errorCode": "RESOURCE_LIMIT_EXCEEDED",
"message": "A timeout occurred when processing the table after 3 updates across 1 iterations. If it continues to happen, please contact your data provider and request them to optimize their table so that it has fewer files."

API:

https://<regionName>.azuredatabricks.net/api/2.0/delta-sharing/metastores/<MetastoreID>/shares/<Shar...

[MaxGendu]

Hi Data_Analytics1

Use Optimize on your delta tables. Refer https://docs.databricks.com/en/sql/language-manual/delta-optimize.html

[Data_Analytics1]

Hi @MaxGendu ,

I executed OPTIMIZE command on the delta table but it only optimizes or merge the data files of current snapshot. Version data files are not get affected or optimized by this OPTIMIZE command.

Even the specific version is only having one data file so even after optimizing it will still be the same.