- 9237 Views
- 2 replies
- 5 kudos
Hi All,I want to run an ETL pipeline in a sequential way in my DB notebook. If I run it without resetting the Spark session or restarting the cluster I am getting a data frame key error. I think this might be because of the Spark cache because If I r...
- 9237 Views
- 2 replies
- 5 kudos
Latest Reply
Is there a solution to the above problem? I also would like to restart SparkSession to free my cluster's resources, but when callingspark.stop()the notebook automatically detach and the following error occurs:The spark context has stopped and the dri...
1 More Replies
- 4173 Views
- 4 replies
- 0 kudos
Databricks docs here:https://docs.databricks.com/notebooks/notebook-isolation.htmlstate that "Every notebook attached to a cluster has a pre-defined variable named spark that represents a SparkSession." What if 2 users run the same notebook on the sa...
- 4173 Views
- 4 replies
- 0 kudos
Latest Reply
The spark session is isolated at the notebook level and is not isolated at the user level. So, two users accessing the same notebook will be using the same spark session
3 More Replies
- 12404 Views
- 4 replies
- 5 kudos
I want to use the same spark session which created in one notebook and need to be used in another notebook in across same environment, Example, if some of the (variable)object got initialized in the first notebook, i need to use the same object in t...
- 12404 Views
- 4 replies
- 5 kudos
Latest Reply
You can use %run and then use the location of the notebook - %run "/folder/notebookname"
3 More Replies
by
jm99
• New Contributor III
- 4305 Views
- 1 replies
- 1 kudos
Most python examples show the structure of the foreachBatch method as:def foreachBatchFunc(batchDF, batchId):
batchDF.createOrReplaceTempView('viewName')
(
batchDF
._jdf.sparkSession()
.sql(
...
- 4305 Views
- 1 replies
- 1 kudos
Latest Reply
Just found a solution...Need to convert the Java Dataframe (jdf) to a DataFramefrom pyspark import sql
def batchFunc(batchDF, batchId):
batchDF.createOrReplaceTempView('viewName')
sparkSession = batchDF._jdf.sparkSession()
resJdf = sparkSes...
- 11458 Views
- 1 replies
- 1 kudos
I'm getting the error...AttributeError: 'SparkSession' object has no attribute '_wrapped'---------------------------------------------------------------------------AttributeError Traceback (most recent call last)<command-2311820097584616> in <cell li...
- 11458 Views
- 1 replies
- 1 kudos
Latest Reply
this can happen in 10X version try to use 7.3 LTS and share your observationand if it not working there try to create init script and load it to your databricks cluster so whenever your machine go up you can get advantage of that library because some...
