cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Retry api request if fails

Braxx
Contributor II

I have a simple API request to query a table and retrive data, which are then suited into a dataframe. May happened, it fails due to different reasons. How to retry it for let's say 5 times when any kind of error takes place?

Here is an api request:

df  = spark.createDataFrame(sf.bulk.MyTable.query("select Id,  Status from MyTable "))

1 REPLY 1

Manoj
Contributor II

@Bartosz Wachocki​ ,

Use timeout, retry interval ,recursion and exception handling

pseudo code below

timeout = 300

def exec_query(query,timeout):

try:

df = spark.createDataFrame(sf.bulk.MyTable.query(query))

except:

if timeout > 0 :

sleep(60)

exec_query(query)

timeout = timeout - 60

else:

print("Timeout")

break;

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group