cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How do I create spark.sql.session.SparkSession?

valjas
New Contributor III

When I create a session n Databricks it is defaulting to spark.sql.connect.session.SparkSession. How can I connect to spark with out spark connect?

3 REPLIES 3

MichTalebzadeh
Valued Contributor

The Spark Session is already created for by the Databricks environment. However you can create your own

from pyspark.sql import SparkSession
# Initialize Spark session
myspark = SparkSession.builder.appName("YourAppName").getOrCreate()

# Create a sample DataFrame for streaming
data = [("Alice", 1), ("Bob", 2), ("Charlie", 3)]
columns = ["name", "value"]
df = myspark.createDataFrame(data, columns)
df.show(truncate=False)

output
+-------+-----+
|name   |value|
+-------+-----+
|Alice  |1    |
|Bob    |2    |
|Charlie|3    |
+-------+-----+

 

Mich Talebzadeh | Technologist | Data | Generative AI | Financial Fraud
London
United Kingdom

view my Linkedin profile



https://en.everybodywiki.com/Mich_Talebzadeh



Disclaimer: The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner Von Braun)".

Even when I create my own session, it still uses spark connect.

valjas_0-1710509955471.png

 

miguel_ortiz
New Contributor II

Is there any solution to this? Pandera, Evidently and Ydata Profiling break because they don't speak a sql.connect session object. They expect a spark.sql.session.SparkSession it's very frustrating not being to use any of these libraries with the newest DBR :(.  

This post becomes misleading: https://www.databricks.com/blog/2023/04/03/pandas-profiling-now-supports-apache-spark.html

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now