Databricks app giving 'upstream request timeout '

[Rakeshch007]

Hello all,

We are developing an app which is based on flask, which is used to download logs from databricks dbfs location. For this useful case we are using databricks inbuilt App feature to deploy our app.

While we pass a smaller file it is getting downloaded without any error, but if we pass a bigger file like 300-400 mb it is giving 'upstream request timeout' after exactly one minute. But when we ran it in local it is working fine.

So we think it may be because of proxy timeout. But we don't have access to our databricks proxy. How can we increase this timeout for the proxy within the app or using env variables.

Please help me with this

[Isi]

Hey!

It looks like the issue you’re facing might be related to the proxy timeout when downloading large files from DBFS. Since modifying the proxy settings might not be an option, there are a couple of alternative approaches you could consider to mitigate this issue.

First, instead of downloading the entire file at once, you can try downloading it in smaller chunks to avoid hitting the request timeout limit. By implementing chunked downloads, you can bypass proxy-imposed limits and improve reliability.

Another effective approach is to store logs directly in an external cloud storage solution, such as AWS S3, Azure Blob Storage, or Google Cloud Storage, instead of DBFS. Databricks allows you to configure cluster logging to automatically save logs to these cloud storage solutions, making them easier to access and download without relying on DBFS.

🙂