cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to import pyspark in python shell?

Kaniz
Community Manager
Community Manager
 
2 REPLIES 2

dazfuller
Contributor III

Basic process is this (assuming you're using a python virtual environment and have activated it).

(.venv) C:\source\project> python -m pip install pyspark
(.venv) C:\source\project> python
 
>>> from pyspark.sql import SparkSession
>>> spark = SparkSession.builder.appName("example").master("local[*]").getOrCreate()

If you want a specific version of Spark then you need to specify it when install pyspark, such as:

python -m pip install pyspark==3.0.1

Kaniz
Community Manager
Community Manager

Thank you so much @Darren Fullerโ€‹ .

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.