topic Hitting 504 Error with Databricks Delta Share open sharing protocol in Data Engineering

Hitting 504 Error with Databricks Delta Share open sharing protocol

nbrisson — Thu, 27 Mar 2025 22:32:30 GMT

When trying to query a particularly large table that my team has been given access to in a share (we are querying the table using a profile file with bearer token), we are continually hitting the following error:

io.delta.sharing.client.util.UnexpectedHttpStatuss: HTTP request failed with status: HTTP/1.1 504 Gateway Timeout {"error_code":"DEADLINE_EXCEEDED"}

Are there limitations on the databricks delta share server for the amount of data you can query via a share?

Re: Hitting 504 Error with Databricks Delta Share open sharing protocol

nbrisson — Thu, 27 Mar 2025 22:39:39 GMT

As a note we are using the deltaSharing data source like so:

spark.read.format("deltaSharing")

Re: Hitting 504 Error with Databricks Delta Share open sharing protocol

SP_6721 — Mon, 07 Apr 2025 11:11:56 GMT

Hi @nbrisson

This could be due to large metadata in the Delta table being queried via Delta Sharing. There are some known limitations, you can refer to these docs for more details:

And try to run OPTIMIZE and VACUUM command on the table to reduce metadata size and also you can use partitioning to reduce the number of files.