I have tried this solution but problem with this solution is that new query details will not be available until we are preloading the data again. I am thinking of using pagination by using attributes page_token,next_page_token,page_size and loop through it until we find the query. This might not exceed concurrent API Limit. I need to just verify it with databricks team.