jacovangelder
Databricks MVP

You can do so by adding your own dbutils function in your py file:

 

def get_dbutils():
    """
    This is to make your local env (and flake  happy."""
    from pyspark.sql import SparkSession

    spark = SparkSession.getActiveSession()
    if spark.conf.get("spark.databricks.service.client.enabled") == "true":
        from pyspark.dbutils import DBUtils

        return DBUtils(spark)
    else:
        import IPython

        return IPython.get_ipython().user_ns["dbutils"]

 


And then run the notebook from the py file using the following way:

 

get_dbutils().notebook.run('path/to/notebook', <timeout_in_seconds>) 

 

Good luck!