Long-running Python http POST hangs

[Johan_Van_Noten]

As one of the steps in my data engineering pipeline, I need to perform a POST request to a http (not -s) server.
This all works fine, except for the situation described below: it then hangs indefinitely.

Environment:

• Azure Databricks Runtime 13.3 LTS
• Python 3.10.12
• Executing from a notebook

Scenario:

• Some example upload of a (big) file.

headers = {"Content-Type": f"{mime_type}"}
chunk_size = 1024*1024
response = requests.post(
	destination_repo_url, 
	headers=headers,
	auth=auth,
	timeout=10,
	data=(chunk for chunk in iter(lambda: source_file.read(chunk_size), b"")))
response.raise_for_status()​

(Obviously the timeout is to be chosen, but whatever we choose, behavior is identical)

• Example without file upload complexity:

response = requests.post(
   destination_REST_URL_triggering_long_operation,
   auth=auth)​

Expected behavior:

• Operation takes the required time (e.g. 5 minutes), then completes.
  The real transfer time is limited, the server's processing is long, but finally should complete without issues.
• One would expect no difference in behaviour between a local Python and the one running on the Azure Databricks cluster, or a clear motivation why it would behave differently, and how to avoid it.

Observed behavior:

• Operation never ends on client side (server side completes as usual).
• The provided timeout doesn't make any difference, while you would expect the read timeout to trigger since nothing is received from the server during its long postprocessing.

Alternatives tried:

• curl as a subprocess works as expected.
  • You see the upload taking place (in case of upload).
  • Then logs multiple lines with no transfer.
  • Then completes correctly after the server sends back its 204.
• Trying from local system ( so not Databricks), you see using Wireshark:
  • POST operation is invoked
  • Data is sent (in case of the file transfer) and completes
  • Server is postprocessing
  • If timeout < processing time
    • Client gives up on ReadTimeout.
      This is expected, normal behavior. To avoid this, set read timeout big enough.
    • Server continues its work and completes it correctly (but doesn't report it anymore because of connection closure by client).
  • Else
    • Server replies the usual 204 once completed
    • Client completes the blocking post request normally.
• I don't have the server's implementation under control, so I can't change that.