Hi Team,
I have updated spark version from 3.3.2 to 3.5.0 and switched to Databricks 15.4 LTS from 12.2 LTS so as to get Spark 3.5 version on the Databricks compute. We have moved from uploading libraries on DBFS to uploading libraries to Volumes as 15.4 does not support DBFS.
We are running Spark Jar task which has libraries installed from Volumes. In the jar task, I'm executing some GRPC call to an external server from our spark job which are not reaching the target server. There are no error logs on Databricks cluster. But when running the same GRPC calls as a simple java main program from Databricks cluster using web terminal, it is working as expected. The calls are reaching the target GRPC server.
The jar task gets stuck in running state without throwing any error. Upon cancelling the jar task, we get CANCELLED: Thread interrupted error.
Below are my cluster details
Access Mode: Single User
Spark Config: spark.driver.extraJavaOptions -DisGrpcEnabled=true -Dengine=spark -DtypeOfApplication=structuredstreaming -Djava.io.tmpdir=/tmp -DPYTHON_PATH=/opt/MyProject/lib/pyspark_libs/py4j-src.zip:/opt/MyProject/lib/pyspark_libs/pyspark.zip:/usr/lib/pyspark/py4j-src.zip:/usr/lib/pyspark/pyspark.zip:py4j-src.zip:pyspark.zip -DlogLevel=DEBUG -DgrpcServerHost=sample.grpc.com -DgrpcServerPort=15005 -DuserToken=O8XFEqjHrSPB7cKGu7KQkhJnFxCORVJ/KXE3bLXZrnRbV+Esssmptw== -DpipelineTenantId=1394 -Dsax.zookeeper.root=/sax -DaccountId=134
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.classesToRegister org.apache.spark.sql.SparkSession
spark.databricks.delta.preview.enabled true
spark.databricks.delta.logStore.crossCloud.fatal false
spark.sql.extensions io.databricks.spark.sql.metastore.DefaultV2SessionCatalog
spark.sql.jsonGenerator.ignoreNullFields false
spark.databricks.delta.formatCheck.enabled false
spark.databricks.service.server.enabled true
spark.master local[*,4]
spark.databricks.cluster.profile singleNode
spark.executor.extraJavaOptions -DisGrpcEnabled=true -Dengine=spark -DtypeOfApplication=structuredstreaming -Djava.io.tmpdir=/tmp -DlogLevel=DEBUG -DgrpcServerHost=sample.grpc.com -DgrpcServerPort=15005 -DuserToken=O8XFEqjHrSPB7cKGu7KQkhJnFxCORVJ/KXE3bLXZrnRbV+Esssmptw== -DpipelineTenantId=1394 -Dsax.zookeeper.root=/sax -DaccountId=134
spark.sql.catalogImplementation in-memory
spark.databricks.acl.sql.enabled true
spark.python.trustedDataTransfer.enabled true
Policy: unrestricted
Databricks Runtime Version: 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)
Libraries are installed from Volumes
Main Program:
public static void main(String[] args) throws InterruptedException, SSLException {
System.out.println("Running grpc client code");
String authToken = "tokenValue";
System.out.println("authToken: " + authToken);
String host = "sample.grpc.com";
System.out.println("host: " + host);
int port = 15005;
System.out.println("port: " + port);
ManagedChannel channel = ManagedChannelBuilder.forAddress(host, port).useTransportSecurity().intercept(new AuthClientInterceptor(authToken))
.enableRetry().maxRetryAttempts(2).build();
System.out.println("channel created");
GrpcServiceDefinitionGrpc.GrpcServiceDefinitionBlockingStub grpcStub = GrpcServiceDefinitionGrpc.newBlockingStub(channel);
System.out.println("stub created");
DynamicMethodRequest request = DynamicMethodRequest.newBuilder().setMethodName("getConfigurations").build();
System.out.println("request created");
DynamicMethodResponse res = grpcStub.invokeDynamicMethod(request);
System.out.println("res : " + res.getResponse());
System.out.println("toStringUtf8 : " + res.getResponse().toStringUtf8());
System.out.println("getResult : " + res.getResult());
channel.shutdown();
}DynamicMethodRequest extends
com.google.protobuf.GeneratedMessageV3 implements
DynamicMethodRequestOrBuilder
@javax.annotation.Generated(
value = "by gRPC proto compiler (version 1.56.1)",
comments = "Source: gRPC.proto")
@io.gathr.grpc.stub.annotations.GrpcGenerated
public final class GrpcServiceDefinitionGrpc
