SOAP API - Connection

[AlexPedurand]

Hello

We have a workflow in our team to perform usual monthly tasks to be ran on the first working day of the month.

Each of the ~20 users will run a clone of this workflow most likely all around the same time but with different options. Because we don't have access to Job-Compute, it runs on a few All-Purpose Computes shared across users.

The first step of this workflow consists in downloading data using a SOAP API (wrapped in a R Package). Since two months, we observed a significant degradation in performance of this task, going from ~5min to ~10 min, if it ever finishes.

It feels like the network now can't handle the possibly concurrent calls to the API. Restarting a cluster and organizing the users in a queue solves the issue but is far from being optimal.

Any recommendations for improvements here ?

Thanks

[feiyun0112]

maybe you can set a lock before call SOAP API

python - Using a Lock with redis-py - Stack Overflow

[Kaniz]

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀