https://community.databricks.com/t5/data-engineering/soap-api-connection/m-p/59087#M31336 <P>maybe you can set a lock before call SOAP API</P><P><A href="https://stackoverflow.com/questions/34171297/using-a-lock-with-redis-py" target="_blank">python - Using a Lock with redis-py - Stack Overflow</A></P> Sat, 03 Feb 2024 02:59:13 GMT feiyun0112 2024-02-03T02:59:13Z https://community.databricks.com/t5/data-engineering/soap-api-connection/m-p/59067#M31327 <P>Hello</P><P>We have a workflow in our team to perform usual monthly tasks to be ran on the first working day of the month.</P><P>Each of the ~20 users will run a clone of this workflow most likely all around the same time but with different options. Because we don't have access to Job-Compute, it runs on a few All-Purpose Computes shared across users.</P><P>The first step of this workflow consists in downloading data using a SOAP API (wrapped in a R Package). Since two months, we observed a significant degradation in performance of this task, going from ~5min to ~10 min, if it ever finishes.</P><P>It feels like the network now can't handle the possibly concurrent calls to the API. Restarting a cluster and organizing the users in a queue solves the issue but is far from being optimal.</P><P>Any recommendations for improvements here ?</P><P>Thanks</P><P>&nbsp;</P> Fri, 02 Feb 2024 15:36:32 GMT https://community.databricks.com/t5/data-engineering/soap-api-connection/m-p/59067#M31327 AlexPedurand 2024-02-02T15:36:32Z https://community.databricks.com/t5/data-engineering/soap-api-connection/m-p/59087#M31336 <P>maybe you can set a lock before call SOAP API</P><P><A href="https://stackoverflow.com/questions/34171297/using-a-lock-with-redis-py" target="_blank">python - Using a Lock with redis-py - Stack Overflow</A></P> Sat, 03 Feb 2024 02:59:13 GMT https://community.databricks.com/t5/data-engineering/soap-api-connection/m-p/59087#M31336 feiyun0112 2024-02-03T02:59:13Z