Databricks SQL Statement Execution API

RamozanbekS · ‎07-25-2023

I'm trying to follow the example provided here https://github.com/databricks-demos/dbsql-rest-api/blob/main/python/external_links.py

It fails when it comes to downloading the data chunks. The statement status turns from SUCCEEDED to CLOSED right away without giving me time to download the data.

Here's the error message I get

<Response [404]>
b'{"error_code":"NOT_FOUND","message":"The result of the statement 01ee2ae9-da85-1e4f-a760-0abb04e34beb was not found.","details":[{"@type":"type.googleapis.com/google.rpc.ResourceInfo","resource_type":"result","resource_name":"01ee2ae9-da85-1e4f-a760-0abb04e34beb","owner":"","description":"the statement state should be SUCCEEDED before you can start fetching the result"}]}'

How can I configure my queries or my sql warehouse so that my queries have at some time to be downloaded?

RamozanbekS · ‎07-25-2023

It turns out that if the response is small and can fit 16mb limit, then status check will also provide single external link to download the data.

So I need a condition here. Maybe even something like this

if len(chunks) == 1:
    external_url = response.json()['result']['external_links'][0]['external_link']

View solution in original post