topic Re: How do I create spark.sql.session.SparkSession? in Get Started Discussions

How do I create spark.sql.session.SparkSession?

valjas — Fri, 15 Mar 2024 12:06:01 GMT

When I create a session n Databricks it is defaulting to spark.sql.connect.session.SparkSession. How can I connect to spark with out spark connect?

Re: How do I create spark.sql.session.SparkSession?

MichTalebzadeh — Fri, 15 Mar 2024 13:22:36 GMT

The Spark Session is already created for by the Databricks environment. However you can create your own

from pyspark.sql import SparkSession # Initialize Spark session myspark = SparkSession.builder.appName("YourAppName").getOrCreate() # Create a sample DataFrame for streaming data = [("Alice", 1), ("Bob", 2), ("Charlie", 3)] columns = ["name", "value"] df = myspark.createDataFrame(data, columns) df.show(truncate=False) output +-------+-----+ |name |value| +-------+-----+ |Alice |1 | |Bob |2 | |Charlie|3 | +-------+-----+

Re: How do I create spark.sql.session.SparkSession?

valjas — Fri, 15 Mar 2024 13:37:51 GMT

Even when I create my own session, it still uses spark connect.

Re: How do I create spark.sql.session.SparkSession?

miguel_ortiz — Wed, 14 Aug 2024 04:15:25 GMT

Is there any solution to this? Pandera, Evidently and Ydata Profiling break because they don't speak a sql.connect session object. They expect a spark.sql.session.SparkSession it's very frustrating not being to use any of these libraries with the newest DBR :(.

This post becomes misleading: https://www.databricks.com/blog/2023/04/03/pandas-profiling-now-supports-apache-spark.html