ffmpeg does not run in a distributed manner afaik. So it will run on the driver.

If you want to run ffmpeg in parallel, you would have to find some distributed version of it. I have found some github projects but they seem pretty dead.

Perhaps you can take a look into OpenCV.

Some time ago I spoke with people who do AI (computer vision) on video, and they used Nvidia Deepstream. Might be worth looking into.