topic writing into delta table using databricks connect in Warehousing & Analytics

writing into delta table using databricks connect

MightyMasdo — Mon, 26 Aug 2024 12:07:15 GMT

Hello we are currently facing a challenge with writing data from a local machine to Delta tables. While small data loads (e.g., 100 Rows) work without any issues, attempting to write larger batches (around 1,000 Rows) results in an exception. the issue started to show up since last week, we were able to write tables to databricks on 10000 batch size. Unfortunately, The error message isn't very descriptive, making it difficult to pinpoint the cause.
Has anyone else experienced a similar issue, or does anyone have insights into potential solutions? We would greatly appreciate any guidance on best practices or troubleshooting steps for writing larger datasets to Delta tables remotely.
Thank you for your help! this is the Error we get

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "INVALID_PARAMETER_VALUE: Invalid compute mode headers. Please pass X-Databricks-Cluster-Id for using Spark Connect
with classic compute or pass X-Databricks-Session-Id for using Spark Connect with
serverless compute. (requestId= XXXX)"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"INVALID_PARAMETER_VALUE: Invalid compute mode headers. Please pass X-Databricks-Cluster-Id for using Spark Connect\nwith classic compute or pass X-Databricks-Session-Id for using Spark Connect with\nserverless compute. (requestId=98ed3355-9a30-45bf-82c9-10ec05f9336e)", grpc_status:3, created_time:"2024-08-26T11:10:18.3724432+00:00"}"
>

Re: writing into delta table using databricks connect

Mok9rQWafV — Fri, 27 Sep 2024 07:04:58 GMT

Hi,

I am experiencing the same behavior on LTS 13.3 and LTS 14.0. Any idea what could be happening?

Re: writing into delta table using databricks connect

AnatolBeck — Mon, 02 Dec 2024 13:09:31 GMT

Also got similar logs since the beginning of the week with Databricks 13.3 LTS.
```

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "INVALID_PARAMETER_VALUE: Invalid compute mode headers. Please pass X-Databricks-Cluster-Id for using Spark Connect
with classic compute or pass X-Databricks-Session-Id for using Spark Connect with
serverless compute. (requestId=XXXX)"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"INVALID_PARAMETER_VALUE: Invalid compute mode headers. Please pass X-Databricks-Cluster-Id for using Spark Connect\nwith classic compute or pass X-Databricks-Session-Id for using Spark Connect with\nserverless compute. (requestId=XXXX)", grpc_status:3, created_time:"2024-12-02T14:04:12.790228698+01:00"}"

```

Re: writing into delta table using databricks connect

AnatolBeck — Mon, 02 Dec 2024 13:47:30 GMT

I can however confirm that writing the data in chunks (playing around with the size) can resolve this error.

The error message is on the one hand completely random and unrelated to real issue and on the other hand the chunk size that allows to write without errors seems to change over time.

This needs a review.

Re: writing into delta table using databricks connect

vr — Mon, 30 Dec 2024 19:41:08 GMT

Hi,

Can you provide a sample code to reproduce the issue?

Re: writing into delta table using databricks connect

chaedke — Mon, 07 Apr 2025 18:26:30 GMT

Any resolution found for this issue? I'm experiencing the same error now myself.

Re: writing into delta table using databricks connect

vr — Mon, 07 Apr 2025 22:55:55 GMT

Code?