cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

spark context in databricks

Data_Engineer3
Contributor III

Hi @all,

In Azure Databricks,
I am using structured streaming for each batch functionality, in one of the functions I am creating tempview with pyspark dataframe (*Not GlobalTempView) and trying to access the same temp view by using spark.sql functionality.

But I am getting the error temp view not found, seems like the temp view is not created in the same spark context.

I want to create the temp view by using pyspark dataframe and access the tempview using spark sql function in structuring streaming. Could you please share the steps to create tempview and access within same databricks notebook.

is spark keyword and spark.sql will have different spark context within same databricks notebook? or both will have same context.

 

#databricks #pyspark #sparksql

1 ACCEPTED SOLUTION

Accepted Solutions

Lakshay
Esteemed Contributor

Do you face this issue without spark streaming as well? Also, could you share a minimal repo code preferably without streaming?

View solution in original post

2 REPLIES 2

Lakshay
Esteemed Contributor

Do you face this issue without spark streaming as well? Also, could you share a minimal repo code preferably without streaming?

Hi @Lakshay 

Without structured streaming, I can able to create the tempview and make use of it in the same notebook.
But for the same, when I try to use in structured streaming, it is throwing the error message that view not found.

please refer the below sample code snippet for reference,

write_to_final_table =  (spark.readStream.format('delta').option('ignoreChanges',True).table(f"{delta_table_name}")).writeStream.queryName(f"{query_name}").format("org.elasticsearch.spark.sql").trigger(processingTime=f'1 minutes').outputMode("append").foreachBatch(process_micro_batch).option("checkpointLocation",checkpointdirectory_path).option("mergeSchema", "true").option("failOnDataLoss", "false").start()


def process_micro_batch(micro_batch_df, batchId) :
    micro_batch_df.createOrReplaceTempView("temp_view")
    df = spark.sql(f"select * from temp_view")
    return df
 
Here, I am getting error, while reading data from temp_view that temp_view not found error.
 
Thanks

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group