-werners-
Esteemed Contributor III

a record count is very easy: first read the delta table in a DF and then do df.count

How fast: depends on the cluster and the lineage of the dataframe (what transformations are applied to it).

There is no way to tell. But a single node cluster with 4 cores will process 8 threads in parallel I believe.

So depending on the amount of data this will return within a few seconds or half an hour or more.

The out of memory error is weird as a record count is stored in the metadata of the table. So it does not take a lot of memory.

What exactly are you trying to do in your code, because it seems you try do process a lot of data locally, not only a record count.