How to detect if running in a workflow job?

[dollyb]

Hi there,

what's the best way to differentiate in what environment my Spark session is running? Locally I develop with databricks-connect's DatabricksSession, but that doesn't work when running a workflow job which requires SparkSession.getOrCreate(). Right now in the job I'm passing a parameter that the app is reading. Is there another robust way to detected if the app is running on a Databricks cluster or not?

[Kaniz · Accepted solution]

Hi @dollyb, When distinguishing between environments where your Spark session is running, especially when transitioning from local development to a workflow job, it’s essential to ensure robust detection.

Here are some approaches you can consider:

1. Cluster Context Using Notebook Context:

   • In Databricks notebooks, you can use the dbutils.notebook.getContext method to extract information about the attached cluster. This includes details like the cluster ID, workspace domain name, and authentication token¹.
   • By examining the cluster ID or other attributes, you can infer whether the code is running within a Databricks environment.

2. Spark Session Isolation:

   • In Databricks, the SparkSession serves as the entry point for using Spark APIs and configuring runtime settings.
   • By default, Spark session isolation is enabled, meaning each notebook or job gets its own isolated Spark session.
   • You can disable this isolation by setting spark.databricks.session.share to true in the Spark config...².
   • If you find that your code is sharing a Spark session across notebooks or jobs, it’s likely running in a Databricks environment.

3. Check for Streaming Tab in Spark UI:

   • When viewing the Spark UI, look for the “Streaming” tab.
   • If the tab is visible, it indicates that a streaming job is running in the cluster³.
   • Absence of the “Streaming” tab suggests a non-streaming context.

4. Environment-Specific Configuration Parameters:

   • Consider using environment-specific configuration parameters.
   • For example, pass a parameter (as you’re currently doing) that indicates whether the app is running on a Databricks cluster or not.
   • This approach provides flexibility and allows you to adapt to different environments.

Remember that the choice depends on your specific use case and requirements. By combining these methods, you can create a robust mechanism to detect whether your Spark session is running in a Databricks cluster or elsewhere. 🚀

[dollyb]

Thanks, dbutils.notebook.getContext does indeed contain information about the job run.