cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How do I create spark.sql.session.SparkSession?

valjas
New Contributor III

When I create a session n Databricks it is defaulting to spark.sql.connect.session.SparkSession. How can I connect to spark with out spark connect?

3 REPLIES 3

MichTalebzadeh
Valued Contributor

The Spark Session is already created for by the Databricks environment. However you can create your own

from pyspark.sql import SparkSession
# Initialize Spark session
myspark = SparkSession.builder.appName("YourAppName").getOrCreate()

# Create a sample DataFrame for streaming
data = [("Alice", 1), ("Bob", 2), ("Charlie", 3)]
columns = ["name", "value"]
df = myspark.createDataFrame(data, columns)
df.show(truncate=False)

output
+-------+-----+
|name   |value|
+-------+-----+
|Alice  |1    |
|Bob    |2    |
|Charlie|3    |
+-------+-----+

 

Mich Talebzadeh | Technologist | Data | Generative AI | Financial Fraud
London
United Kingdom

view my Linkedin profile



https://en.everybodywiki.com/Mich_Talebzadeh



Disclaimer: The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner Von Braun)".

Even when I create my own session, it still uses spark connect.

valjas_0-1710509955471.png

 

miguel_ortiz
New Contributor II

Is there any solution to this? Pandera, Evidently and Ydata Profiling break because they don't speak a sql.connect session object. They expect a spark.sql.session.SparkSession it's very frustrating not being to use any of these libraries with the newest DBR :(.  

This post becomes misleading: https://www.databricks.com/blog/2023/04/03/pandas-profiling-now-supports-apache-spark.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group