writing into delta table using databricks connect
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-26-2024 05:05 AM - edited 08-26-2024 05:07 AM
Hello we are currently facing a challenge with writing data from a local machine to Delta tables. While small data loads (e.g., 100 Rows) work without any issues, attempting to write larger batches (around 1,000 Rows) results in an exception. the issue started to show up since last week, we were able to write tables to databricks on 10000 batch size. Unfortunately, The error message isn't very descriptive, making it difficult to pinpoint the cause.
Has anyone else experienced a similar issue, or does anyone have insights into potential solutions? We would greatly appreciate any guidance on best practices or troubleshooting steps for writing larger datasets to Delta tables remotely.
Thank you for your help! this is the Error we get
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "INVALID_PARAMETER_VALUE: Invalid compute mode headers. Please pass X-Databricks-Cluster-Id for using Spark Connect
with classic compute or pass X-Databricks-Session-Id for using Spark Connect with
serverless compute. (requestId= XXXX)"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"INVALID_PARAMETER_VALUE: Invalid compute mode headers. Please pass X-Databricks-Cluster-Id for using Spark Connect\nwith classic compute or pass X-Databricks-Session-Id for using Spark Connect with\nserverless compute. (requestId=98ed3355-9a30-45bf-82c9-10ec05f9336e)", grpc_status:3, created_time:"2024-08-26T11:10:18.3724432+00:00"}"
>
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-27-2024 12:04 AM
Hi,
I am experiencing the same behavior on LTS 13.3 and LTS 14.0. Any idea what could be happening?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2024 05:08 AM - edited 12-02-2024 05:09 AM
Also got similar logs since the beginning of the week with Databricks 13.3 LTS.
```
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "INVALID_PARAMETER_VALUE: Invalid compute mode headers. Please pass X-Databricks-Cluster-Id for using Spark Connect
with classic compute or pass X-Databricks-Session-Id for using Spark Connect with
serverless compute. (requestId=XXXX)"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"INVALID_PARAMETER_VALUE: Invalid compute mode headers. Please pass X-Databricks-Cluster-Id for using Spark Connect\nwith classic compute or pass X-Databricks-Session-Id for using Spark Connect with\nserverless compute. (requestId=XXXX)", grpc_status:3, created_time:"2024-12-02T14:04:12.790228698+01:00"}"
```
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-02-2024 05:47 AM
I can however confirm that writing the data in chunks (playing around with the size) can resolve this error.
The error message is on the one hand completely random and unrelated to real issue and on the other hand the chunk size that allows to write without errors seems to change over time.
This needs a review.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Hi,
Can you provide a sample code to reproduce the issue?