topic spark context in databricks in Get Started Discussions

spark context in databricks

Data_Engineer3 — Tue, 30 Jan 2024 11:41:03 GMT

Hi @all,

In Azure Databricks,
I am using structured streaming for each batch functionality, in one of the functions I am creating tempview with pyspark dataframe (*Not GlobalTempView) and trying to access the same temp view by using spark.sql functionality.

But I am getting the error temp view not found, seems like the temp view is not created in the same spark context.

I want to create the temp view by using pyspark dataframe and access the tempview using spark sql function in structuring streaming. Could you please share the steps to create tempview and access within same databricks notebook.

is spark keyword and spark.sql will have different spark context within same databricks notebook? or both will have same context.

#databricks #pyspark #sparksql

Re: spark context in databricks

Lakshay — Tue, 30 Jan 2024 13:11:14 GMT

Do you face this issue without spark streaming as well? Also, could you share a minimal repo code preferably without streaming?

Re: spark context in databricks

Data_Engineer3 — Wed, 31 Jan 2024 04:50:37 GMT

Hi @Lakshay

Without structured streaming, I can able to create the tempview and make use of it in the same notebook.
But for the same, when I try to use in structured streaming, it is throwing the error message that view not found.

please refer the below sample code snippet for reference,

write_to_final_table = (spark.readStream.format('delta').option('ignoreChanges',True).table(f"{delta_table_name}")).writeStream.queryName(f"{query_name}").format("org.elasticsearch.spark.sql").trigger(processingTime=f'1 minutes').outputMode("append").foreachBatch(process_micro_batch).option("checkpointLocation",checkpointdirectory_path).option("mergeSchema", "true").option("failOnDataLoss", "false").start()

def process_micro_batch(micro_batch_df, batchId) :

micro_batch_df.createOrReplaceTempView("temp_view")

df = spark.sql(f"select * from temp_view")

return df

Here, I am getting error, while reading data from temp_view that temp_view not found error.

Thanks