Performance issue while calling Sagemaker Endpoint in pyspark udf
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-30-2023 07:53 AM
Hi,
I have pyspark dataframe which calls pyspark udf which in turn calls sagemaker endpoint. But when dataframe has more rows, endpoint start failing. Also it takes longer to process.
Please suggest how to call sagemaker endpoint from pyspark.
Regards,
Sanjay
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-31-2023 12:01 AM - edited 10-31-2023 12:01 AM
Thank you @Retired_mod for prompt response.
"send batch to sagemaker", you mean sending multiple data records in every sagemaker call? As sagemaker has 60 second timeout, will this not timeout the request if there are multiple records in single call
