this is smart, happy to learn that hierarchical groupings affect the overall performance based on sample size at each hierarchy. 

If OOM: resort to Spark to split across nodes for you.

If small enough at a hierarchy level, prefer to use single-node in-memory operation by pd.df.apply() to accelerate and avoid broadcasting overheads.