topic Re: Delta Sharing CDF API error: "RESOURCE_LIMIT_EXCEEDED" in Get Started Discussions

Delta Sharing CDF API error: "RESOURCE_LIMIT_EXCEEDED"

Data_Analytics1 — Fri, 13 Oct 2023 07:14:42 GMT

Hi,

When attempting to read a particular version from the Databricks Delta Sharing CDF (Change Data Feed) API, even when that version contains only one data file, an error occurs due to a timeout with following message:

"errorCode": "RESOURCE_LIMIT_EXCEEDED",
"message": "A timeout occurred when processing the table after 3 updates across 1 iterations. If it continues to happen, please contact your data provider and request them to optimize their table so that it has fewer files."

API:

https://<regionName>.azuredatabricks.net/api/2.0/delta-sharing/metastores/<MetastoreID>/shares/<ShareName>/schemas/<SchemaName>/tables/<TableName>/changes?startingVersion=40000&endingVersion=40000

Re: Delta Sharing CDF API error: "RESOURCE_LIMIT_EXCEEDED"

MaxGendu — Fri, 13 Oct 2023 08:01:27 GMT

Hi Data_Analytics1

Use Optimize on your delta tables. Refer https://docs.databricks.com/en/sql/language-manual/delta-optimize.html

Re: Delta Sharing CDF API error: "RESOURCE_LIMIT_EXCEEDED"

Data_Analytics1 — Fri, 13 Oct 2023 08:05:30 GMT

Hi @MaxGendu ,

I executed OPTIMIZE command on the delta table but it only optimizes or merge the data files of current snapshot. Version data files are not get affected or optimized by this OPTIMIZE command.

Even the specific version is only having one data file so even after optimizing it will still be the same.