GRPC call are not getting through on Databricks 15.4 LTS

[sahil_s_jain]

Hi Team,

I have updated spark version from 3.3.2 to 3.5.0 and switched to Databricks 15.4 LTS from 12.2 LTS so as to get Spark 3.5 version on the Databricks compute. We have moved from uploading libraries on DBFS to uploading libraries to Volumes as 15.4 does not support DBFS.

We are running Spark Jar task which has libraries installed from Volumes. In the jar task, I'm executing some GRPC call to an external server from our spark job which are not reaching the target server. There are no error logs on Databricks cluster. But when running the same GRPC calls as a simple java main program from Databricks cluster using web terminal, it is working as expected. The calls are reaching the target GRPC server.

The jar task gets stuck in running state without throwing any error. Upon cancelling the jar task, we get CANCELLED: Thread interrupted error.

Below are my cluster details

Access Mode: Single User

Spark Config: spark.driver.extraJavaOptions -DisGrpcEnabled=true -Dengine=spark -DtypeOfApplication=structuredstreaming -Djava.io.tmpdir=/tmp -DPYTHON_PATH=/opt/MyProject/lib/pyspark_libs/py4j-src.zip:/opt/MyProject/lib/pyspark_libs/pyspark.zip:/usr/lib/pyspark/py4j-src.zip:/usr/lib/pyspark/pyspark.zip:py4j-src.zip:pyspark.zip -DlogLevel=DEBUG -DgrpcServerHost=sample.grpc.com -DgrpcServerPort=15005 -DuserToken=O8XFEqjHrSPB7cKGu7KQkhJnFxCORVJ/KXE3bLXZrnRbV+Esssmptw== -DpipelineTenantId=1394 -Dsax.zookeeper.root=/sax -DaccountId=134
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.classesToRegister org.apache.spark.sql.SparkSession
spark.databricks.delta.preview.enabled true
spark.databricks.delta.logStore.crossCloud.fatal false
spark.sql.extensions io.databricks.spark.sql.metastore.DefaultV2SessionCatalog
spark.sql.jsonGenerator.ignoreNullFields false
spark.databricks.delta.formatCheck.enabled false
spark.databricks.service.server.enabled true
spark.master local[*,4]
spark.databricks.cluster.profile singleNode
spark.executor.extraJavaOptions -DisGrpcEnabled=true -Dengine=spark -DtypeOfApplication=structuredstreaming -Djava.io.tmpdir=/tmp -DlogLevel=DEBUG -DgrpcServerHost=sample.grpc.com -DgrpcServerPort=15005 -DuserToken=O8XFEqjHrSPB7cKGu7KQkhJnFxCORVJ/KXE3bLXZrnRbV+Esssmptw== -DpipelineTenantId=1394 -Dsax.zookeeper.root=/sax -DaccountId=134
spark.sql.catalogImplementation in-memory
spark.databricks.acl.sql.enabled true
spark.python.trustedDataTransfer.enabled true

Policy: unrestricted

Databricks Runtime Version: 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)

Libraries are installed from Volumes

Main Program:

public static void main(String[] args) throws InterruptedException, SSLException {
    System.out.println("Running grpc client code");
    String authToken = "tokenValue";
    System.out.println("authToken: " + authToken);
    String host = "sample.grpc.com";
    System.out.println("host: " + host);
    int port = 15005;
    System.out.println("port: " + port);

    ManagedChannel channel = ManagedChannelBuilder.forAddress(host, port).useTransportSecurity().intercept(new AuthClientInterceptor(authToken))
            .enableRetry().maxRetryAttempts(2).build();
    System.out.println("channel created");

    GrpcServiceDefinitionGrpc.GrpcServiceDefinitionBlockingStub grpcStub = GrpcServiceDefinitionGrpc.newBlockingStub(channel);
    System.out.println("stub created");
    DynamicMethodRequest request = DynamicMethodRequest.newBuilder().setMethodName("getConfigurations").build();
    System.out.println("request created");
    DynamicMethodResponse res = grpcStub.invokeDynamicMethod(request);
    System.out.println("res : " + res.getResponse());
    System.out.println("toStringUtf8 : " + res.getResponse().toStringUtf8());
    System.out.println("getResult : " + res.getResult());
    channel.shutdown();
}

DynamicMethodRequest extends
    com.google.protobuf.GeneratedMessageV3 implements
    DynamicMethodRequestOrBuilder

@javax.annotation.Generated(
    value = "by gRPC proto compiler (version 1.56.1)",
    comments = "Source: gRPC.proto")
@io.gathr.grpc.stub.annotations.GrpcGenerated
public final class GrpcServiceDefinitionGrpc

[Walter_C]

Do you have this script set up on the job cluster? What happens if you select an all purpose cluster for this job using the same configuration?

Was the cluster with 12.2 DBR version using the same access mode as the one you are using now?

[sahil_s_jain]

I have tried on both all purpose compute and job compute.

Yes, same access mode was configured in 2.2 LTS.

[Walter_C]

And this was working before is it correct? When the init was hosted in DBFS

[sahil_s_jain]

It is not init script. It is a spark jar task which is executing some GRPC call to an external server from our spark job which are not reaching the target server.

It was working when I was using 12.2 LTS.