Delta Sharing - RESOURCE_LIMIT_EXCEEDED (AddFiles, RemoveFiles)

Leszek — Wed, 03 Apr 2024 06:42:55 GMT

Hi,

I would like to start using Delta Sharing but I would need to check if my tables hit limitations which are in the following article: RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table - Databricks.

Delta Sharing has limits on the metadata size of a shared table.

You are limited to 700k AddFiles actions in the DeltaLog. This is how many active files you can have in a shared Delta table.
You are limited to 100k RemoveFiles actions in the DeltaLog. This is the number of files that have been deleted. This includes files that have been removed by operations like OPTIMIZE and MERGE.

Basically, the question is how can I calculate number of AddFiles / RemoveFiles for Delta Table?

topic Delta Sharing - RESOURCE_LIMIT_EXCEEDED (AddFiles, RemoveFiles) in Data Engineering