dbutils.notebook API: pass data back to caller notebook

[lauraxyz]

Hi all, 

according to this doc, we can pass data back through temp views, DBFS, or JSON data.

However, in my case, i need to pass both a temp view, as well as some metadata in JSON.  is there a way to exit with BOTH a view AND json, something like

dbutils.notebook.exit("my_view_name", json.dumps({
  "status": "OK",
  "table": "my_data"
}))

[SparkJun]

Can we try using a combination of dbutils.notebook.exit() and temporary views? there is a drawback that dbutils.notebook.exit() can only return a single string.

import json
from pyspark.sql import SparkSession

# Create a temporary view
data = spark.range(5).toDF("value")
data.createOrReplaceTempView("my_view_name")

# Create the metadata
metadata = {
    "status": "OK",
    "table": "my_view_name"
}

# Exit the notebook with the JSON string
dbutils.notebook.exit(json.dumps(metadata))

import json
from pyspark.sql import SparkSession

data = spark.range(5).toDF("value")
data.createOrReplaceTempView("my_view_name")

metadata = {
    "status": "OK",
    "table": "my_view_name"
}

dbutils.notebook.exit(json.dumps(metadata))

use this notebook to call it:

import json

# Run the callee notebook and get the result
result = dbutils.notebook.run("path_to_callee_notebook", 60)

# Parse the JSON result
metadata = json.loads(result)

# Access the temporary view
global_temp_db = spark.conf.get("spark.sql.globalTempDatabase")
view_name = metadata["table"]
display(spark.sql(f"SELECT * FROM {global_temp_db}.{view_name}"))

[lauraxyz]

I can have a try and see if it's the same like exit(view_name) in that the view is created in global_temp_db and that the lifecycle is with the job compute.