Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-21-2021 11:45 PM
a record count is very easy: first read the delta table in a DF and then do df.count
How fast: depends on the cluster and the lineage of the dataframe (what transformations are applied to it).
There is no way to tell. But a single node cluster with 4 cores will process 8 threads in parallel I believe.
So depending on the amount of data this will return within a few seconds or half an hour or more.
The out of memory error is weird as a record count is stored in the metadata of the table. So it does not take a lot of memory.
What exactly are you trying to do in your code, because it seems you try do process a lot of data locally, not only a record count.