It turned out to be the BroadcastNestedLoopJoin. Once I reworked my logic to remove that, the performance cleared up.