topic Re: Pyspark installation issue in Data Engineering

Pyspark installation issue

_r_vind1199 — Thu, 14 Apr 2022 18:25:39 GMT

When I try to start pyspark session in pycharm. It throws me this error "RuntimeError("Java gateway process exited before sending its port number"). Could anyone help me to solve this?

Re: Pyspark installation issue

Aashita — Thu, 14 Apr 2022 23:31:37 GMT

@Aravind A , What are the dependencies and versions you installed?

Install pyspark + openjdk

%pip install pyspark==2.4.8
!apt-get install openjdk-8-jdk-headless -qq > /dev/null

Build spark session

from pyspark.sql import SparkSession
spark = SparkSession.builder\
    .appName("Practice”)\
    .getOrCreate()

Note that:

Spark runs on Java 8/11, Scala 2.12/2.13, Python 3.6+ and R 3.5+.

Python 3.6 support is deprecated as of Spark 3.2.0.

Java 8 prior to version 8u201 support is deprecated as of Spark 3.2.0.

For the Scala API, Spark 3.2.1 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).

https://spark.apache.org/docs/latest/

Re: Pyspark installation issue

_r_vind1199 — Fri, 15 Apr 2022 13:47:18 GMT

@Aashita Ramteke , Pyspark version 3.2.1

Re: Pyspark installation issue

Aashita — Wed, 20 Apr 2022 16:36:09 GMT

Kindly replace the version when you install python dependencies from 2.4.8 to 3.2.1.. It should work. Thanks.