topic Re: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number. in Get Started Discussions

[JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

roshan_robert — Mon, 25 Nov 2024 17:04:19 GMT

Hi Team,

In a streamlit app (in databricks) while creating the spark session getting below error, this is happening when running the app via web link.

"[JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number"

Below is the code used in app.py:

import os
import streamlit as st
import pandas as pd
from databricks import sql
import pytz
from datetime import datetime

from pyspark.sql import SparkSession
from pyspark import SparkContext
from pyspark import SparkConf

# Create SparkSession
try:
spark = SparkSession.builder \
.appName("Streamlit App") \
.config("spark.driver.memory", "4g") \
.config("spark.executor.memory", "4g") \
.getOrCreate()
st.write("Spark Session created successfully!")
st.write("Spark Version:", spark.version)
except Exception as e:
st.write(f"Error creating Spark session: {e}")

#Create a sample DataFrame
data = {
'Name': ['Alice', 'Bob', 'Charlie', 'David'],
'Age': [25, 30, 35, 40],
'City': ['New York', 'Los Angeles', 'Chicago', 'Houston']
}
df = pd.DataFrame(data)

# Display the DataFrame in the Streamlit app
st.title('Sample DataFrame')
st.write(df)

# Convert the edited DataFrame to a Spark DataFrame
spark_df = spark.createDataFrame(df)
st.write(spark_df)

Re: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

Walter_C — Mon, 25 Nov 2024 23:54:54 GMT

The error you're encountering, "[JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number", typically occurs when there are issues with the Java configuration or when PySpark is unable to establish a connection with the Java gateway process. Here are some potential solutions to address this issue:

Java Configuration

Java Version: Ensure that you're using a compatible Java version. Spark typically works best with Java 8 or 11. Check your Java version and make sure it's compatible with your Spark version
JAVA_HOME: Verify that the JAVA_HOME environment variable is correctly set and points to a compatible Java installation

Re: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

roshan_robert — Tue, 26 Nov 2024 08:22:24 GMT

I tried setting up JAVA_HOME explicitly, but it did not work. Issue i am facing is when launching the application using the web link, when running the same code in notebook its running fine. Let me know the fix.

Re: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

moisttrickery — Tue, 10 Dec 2024 18:26:45 GMT

Hey @roshan_robert. I am facing a similar issue. We are trying to launch a streamlit app from databricks apps that requires spark but Im getting the same error. I have yet to find a way to configure spark to work from my apps cluster. Did you ever resolve your problem?

Re: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

roshan_robert — Thu, 12 Dec 2024 12:06:16 GMT

I did not get a solution.

Re: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

moisttrickery — Tue, 17 Dec 2024 21:08:27 GMT

We got our app working by using the databricks-connect and creating a spark session on a shared compute cluster separate from the cluster my app runs on. I set the env variable DATABRICKS_CLUSTER_ID to the id of the cluster I want to use. Then in that cluster (and any catalog you need to access) you can set permission for your apps service principal. You can find the name of your apps service principal in the apps overview screen. You can set the SP to have permission to attach. You must also make sure that the version of databricks your cluster is running and the version of databricks-connect you are using are compatible (I just used version 14.* for both).

Re: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

Alberto_Umana — Wed, 18 Dec 2024 13:23:21 GMT

Per looking internally I see that Spark (Context) is not available in Apps

The recommended way would be to use our available SDKs and connect to Clusters/DBSQL. No spark context is available- it’s meant to defer processing to other compute it can connect to via sdk, sql connector, spark connect, jobs, etc