Performance issue while calling Sagemaker Endpoint in pyspark udf

[sanjay]

Hi,

I have pyspark dataframe which calls pyspark udf which in turn calls sagemaker endpoint. But when dataframe has more rows, endpoint start failing. Also it takes longer to process.

Please suggest how to call sagemaker endpoint from pyspark.

Regards,

Sanjay

[sanjay]

Thank you @Retired_mod for prompt response. 

"send batch to sagemaker", you mean sending multiple data records in every sagemaker call? As sagemaker has 60 second timeout, will this not timeout the request if there are multiple records in single call