Access same createorreplacetempview("viewname") by multiple functions.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-19-2022 05:49 AM
I have several functions accessing the same createorreplacetempview("viewname"). Does this cause any issues with multiple functions accessing it in a distributed environment?
def get_data_sql(spark_session, data_frame, data_element):
data_frame.createOrReplaceTempView("data")
return spark_session.sql(
f"""
SELECT id, {data_element} FROM data
"""
)Then I have these two functions calling the above function like this.
def get_key(spark_session, data_frame):
return get_data_sql(
spark_session,data_frame,key
)def get_value(spark_session, data_frame):
return get_data_sql(
spark_session,data_frame,value
)Should I be creating the temporary view in each of the calling functions with a separate name to avoid contention/clash?
My understanding is the once the view is created it lasts for the whole session and there would be no issues with multiple access in a distributed environment.
Labels:
- Labels:
-
Data