Hi @Steffen ,

Perhaps you can classify ids by their data frequency, e.g. low, med, and high-frequency, and process them in different jobs? This will avoid skewness. 

Also as a quick test I would suggest to try to run it on serverless - it will scale up to more ressources which might be enough for the processing.

Let us know how it goes.

Best regards,