topic Retry api request if fails in Data Engineering

Retry api request if fails

Braxx — Thu, 28 Oct 2021 20:10:53 GMT

I have a simple API request to query a table and retrive data, which are then suited into a dataframe. May happened, it fails due to different reasons. How to retry it for let's say 5 times when any kind of error takes place?

Here is an api request:

df  = spark.createDataFrame(sf.bulk.MyTable.query("select Id,  Status from MyTable "))

Re: Retry api request if fails

Manoj — Thu, 28 Oct 2021 22:25:09 GMT

@Bartosz Wachocki ,

Use timeout, retry interval ,recursion and exception handling

pseudo code below

timeout = 300

def exec_query(query,timeout):

try:

df = spark.createDataFrame(sf.bulk.MyTable.query(query))

except:

if timeout > 0 :

sleep(60)

exec_query(query)

timeout = timeout - 60

else:

print("Timeout")

break;