topic Re: dbutils.notebook API: pass data back to caller notebook in Data Engineering

dbutils.notebook API: pass data back to caller notebook

lauraxyz — Fri, 10 Jan 2025 01:29:41 GMT

Hi all,

according to this doc, we can pass data back through temp views, DBFS, or JSON data.

However, in my case, i need to pass both a temp view, as well as some metadata in JSON. is there a way to exit with BOTH a view AND json, something like

dbutils.notebook.exit("my_view_name", json.dumps({
"status": "OK",
"table": "my_data"
}))

Re: dbutils.notebook API: pass data back to caller notebook

SparkJun — Fri, 10 Jan 2025 05:11:25 GMT

Can we try using a combination of dbutils.notebook.exit() and temporary views? there is a drawback that dbutils.notebook.exit() can only return a single string.

import json from pyspark.sql import SparkSession # Create a temporary view data = spark.range(5).toDF("value") data.createOrReplaceTempView("my_view_name") # Create the metadata metadata = { "status": "OK", "table": "my_view_name" } # Exit the notebook with the JSON string dbutils.notebook.exit(json.dumps(metadata))

import json from pyspark.sql import SparkSession data = spark.range(5).toDF("value") data.createOrReplaceTempView("my_view_name") metadata = { "status": "OK", "table": "my_view_name" } dbutils.notebook.exit(json.dumps(metadata))

use this notebook to call it:

import json # Run the callee notebook and get the result result = dbutils.notebook.run("path_to_callee_notebook", 60) # Parse the JSON result metadata = json.loads(result) # Access the temporary view global_temp_db = spark.conf.get("spark.sql.globalTempDatabase") view_name = metadata["table"] display(spark.sql(f"SELECT * FROM {global_temp_db}.{view_name}"))

Re: dbutils.notebook API: pass data back to caller notebook

lauraxyz — Fri, 10 Jan 2025 16:55:19 GMT

I can have a try and see if it's the same like exit(view_name) in that the view is created in global_temp_db and that the lifecycle is with the job compute.