cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Retry api request if fails

Braxx
Contributor II

I have a simple API request to query a table and retrive data, which are then suited into a dataframe. May happened, it fails due to different reasons. How to retry it for let's say 5 times when any kind of error takes place?

Here is an api request:

df  = spark.createDataFrame(sf.bulk.MyTable.query("select Id,  Status from MyTable "))

1 REPLY 1

Manoj
Contributor II

@Bartosz Wachocki​ ,

Use timeout, retry interval ,recursion and exception handling

pseudo code below

timeout = 300

def exec_query(query,timeout):

try:

df = spark.createDataFrame(sf.bulk.MyTable.query(query))

except:

if timeout > 0 :

sleep(60)

exec_query(query)

timeout = timeout - 60

else:

print("Timeout")

break;

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now