Downloading the query result through rest API

[Datagyan]

Hi all i have a specific requirements to download the query result. i have created a table on data bricks using SQL warehouse. I have to fetch the query from a custom UI using data API token. Now I am able to fetch the query, but the problem is what if my table is more than 25 MB then I have to use disposition: external links, so the result I am getting in various chunks and suppose one query result is around 1GB file, then I am getting around 250+ chunks. Now I have to download these 250 files separately, but my requirement is to get only one file. What is the solution so I can get only one file do I need to merge only there is no such other option?

and yes we have to use only sql warehouse compute compute

[Vasireddy]

Hey @Datagyan ,

If your query result is larger than 25MB, Databricks automatically uses disposition=EXTERNAL_LINKS, which returns the result in multiple chunked files(external links).Currently, there's no option to get a single file directly from the SQL Warehouse API.

To get the result as a single file, you need to:
--> Programmatically download all the external links (e.g., using Python), and
--> Merge them into one final file (CSV, JSON, or Parquet).

Since only SQL Warehouse compute is allowed in this scenario, options like writing to DBFS or cloud storage via notebooks are not applicable.