Access same createorreplacetempview("viewname") by multiple functions.

vk217 — Mon, 19 Dec 2022 13:49:25 GMT

I have several functions accessing the same createorreplacetempview("viewname"). Does this cause any issues with multiple functions accessing it in a distributed environment?

def get_data_sql(spark_session, data_frame, data_element):
        data_frame.createOrReplaceTempView("data")
        return spark_session.sql(
        f"""
        SELECT id, {data_element} FROM  data
        """
        )

Then I have these two functions calling the above function like this.

def get_key(spark_session, data_frame):
        return  get_data_sql(
               spark_session,data_frame,key
        )

def get_value(spark_session, data_frame):
        return  get_data_sql(
               spark_session,data_frame,value
        )

Should I be creating the temporary view in each of the calling functions with a separate name to avoid contention/clash?

My understanding is the once the view is created it lasts for the whole session and there would be no issues with multiple access in a distributed environment.

Re: Access same createorreplacetempview("viewname") by multiple functions.

Aviral-Bhardwaj — Tue, 20 Dec 2022 14:06:47 GMT

there is two type of views

one is global view - it will be available for whole cluster and notebook but it will removed after cluster restart

and another is Temp view- that will be available for only notebook level, and other notebook will not able to see that.

I hope you will get little bit idea on this

Thanks

Aviral Bhardwaj

topic Re: Access same createorreplacetempview("viewname") by multiple functions. in Data Engineering

Access same createorreplacetempview("viewname") by multiple functions.

Re: Access same createorreplacetempview("viewname") by multiple functions.