Question: Since I'm using Databricks Free Edition that uses only serverless cluster, I cannot use spark.sql.globalTempDatabase in my code below. What's an alternative solution for the Caller_Notebook below. Following error occurred in the second line of this notebook: globalTempDatabase is not available in serverless compute environment.

I'm working on passing structured data between two notebooks. In the following code, data from called notebook is supposed to return data (1 column with 5 rows) to a caller notebook:

Called_Notebook:

spark.range(5).toDF("MyCol").createOrReplaceTempView("MyView")
dbutils.notebook.exit("my_data")

Caller_Notebook:

returned_table = dbutils.notebook.run("./Called_Notebook", 60)
My_temp_db = spark.conf.get("spark.sql.globalTempDatabase")
display(table(My_temp_db + "." + returned_table))