How do I create spark.sql.session.SparkSession?

[valjas]

When I create a session n Databricks it is defaulting to spark.sql.connect.session.SparkSession. How can I connect to spark with out spark connect?

[MichTalebzadeh]

The Spark Session is already created for by the Databricks environment. However you can create your own

from pyspark.sql import SparkSession
# Initialize Spark session
myspark = SparkSession.builder.appName("YourAppName").getOrCreate()

# Create a sample DataFrame for streaming
data = [("Alice", 1), ("Bob", 2), ("Charlie", 3)]
columns = ["name", "value"]
df = myspark.createDataFrame(data, columns)
df.show(truncate=False)

output
+-------+-----+
|name   |value|
+-------+-----+
|Alice  |1    |
|Bob    |2    |
|Charlie|3    |
+-------+-----+

[valjas]

Even when I create my own session, it still uses spark connect.

[miguel_ortiz]

Is there any solution to this? Pandera, Evidently and Ydata Profiling break because they don't speak a sql.connect session object. They expect a spark.sql.session.SparkSession it's very frustrating not being to use any of these libraries with the newest DBR :(.  

This post becomes misleading: https://www.databricks.com/blog/2023/04/03/pandas-profiling-now-supports-apache-spark.html