topic Re: Using spark jars using databricks-connect>=13.0 in Data Engineering

Using spark jars using databricks-connect>=13.0

Lazloo — Tue, 05 Dec 2023 12:25:10 GMT

With the newest version of databricks-connect, I cannot configure the extra jars I want to use. In the older version, I did that via

spark = SparkSession.builder.appName('DataFrame').\
        config('spark.jars.packages','org.apache.spark:spark-avro_2.12:3.3.0').getOrCreate()

How can I configure this with databricks-connect>=13.0?

Re: Using spark jars using databricks-connect>=13.0

ayush19 — Thu, 18 Jul 2024 07:21:05 GMT

Hello, I have a question on this. I have a jar file which is installed on my Databricks cluster. Do you happen to know how can I access the spark session running in databricks within my code (which later turns into jar)

I have tried using SparkSession.active and SparkSession.getActiveSession but I get error that these methods are not supported.