Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-26-2023 03:26 AM - edited 08-26-2023 03:28 AM
I have the same issue, but with databricks-connect==13.2.1
Code to reproduce:
col_num = 49
data = [tuple([f'val_{n}' for n in range(1, col_num + 1)])]
df = spark.createDataFrame(data=data)
for i in range(1, len(data[0]) + 1):
df = df.withColumnRenamed(f'_{i}', f'col_{i}')
df.printSchema()The error is a bit different though:
pyspark.errors.exceptions.connect.SparkConnectGrpcException: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = ""
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2023-08-26T12:06:04.41660107+02:00", grpc_status:2, grpc_message:""}"
>And if col_num == 48, everytning is fine. If col_num > 49, I will get a segfault without any error message.