Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-10-2024 10:22 PM - edited 07-10-2024 10:23 PM
You can do so by adding your own dbutils function in your py file:
def get_dbutils():
"""
This is to make your local env (and flake happy."""
from pyspark.sql import SparkSession
spark = SparkSession.getActiveSession()
if spark.conf.get("spark.databricks.service.client.enabled") == "true":
from pyspark.dbutils import DBUtils
return DBUtils(spark)
else:
import IPython
return IPython.get_ipython().user_ns["dbutils"]
And then run the notebook from the py file using the following way:
get_dbutils().notebook.run('path/to/notebook', <timeout_in_seconds>)
Good luck!