Downloading the query result through rest API

Datagyan — Thu, 29 May 2025 09:23:31 GMT

Hi all i have a specific requirements to download the query result. i have created a table on data bricks using SQL warehouse. I have to fetch the query from a custom UI using data API token. Now I am able to fetch the query, but the problem is what if my table is more than 25 MB then I have to use disposition: external links, so the result I am getting in various chunks and suppose one query result is around 1GB file, then I am getting around 250+ chunks. Now I have to download these 250 files separately, but my requirement is to get only one file. What is the solution so I can get only one file do I need to merge only there is no such other option?

and yes we have to use only sql warehouse compute compute

Re: Downloading the query result through rest API

HariSankar — Fri, 30 May 2025 15:03:54 GMT

Hey @Datagyan ,

If your query result is larger than 25MB, Databricks automatically uses disposition=EXTERNAL_LINKS, which returns the result in multiple chunked files(external links).Currently, there's no option to get a single file directly from the SQL Warehouse API.

To get the result as a single file, you need to:
--> Programmatically download all the external links (e.g., using Python), and
--> Merge them into one final file (CSV, JSON, or Parquet).

Since only SQL Warehouse compute is allowed in this scenario, options like writing to DBFS or cloud storage via notebooks are not applicable.

topic Downloading the query result through rest API in Data Engineering

Downloading the query result through rest API

Re: Downloading the query result through rest API