Hey @divyab7 

Sorry, now I understand better what you actually need. I got confused at first and thought you only wanted to access the parameters you pass through Airflow.

I think the dynamic identifiers that Databricks generates at runtime (like run IDs) are not injected there automatically.

I have been thinking in a way to get them without using dbutils is:

• Job id  →  you can extract it from spark.conf.get("spark.databricks.clusterUsageTags.clusterName"), which has a value like job-<job_id>-run-<task_run_id>.

• Job run ID → once you have the job_id, you can call the Databricks Jobs API and retrieve the job_run_id.

This approach should work, but I agree it’s not very straightforward. Databricks could definitely make it easier to expose these values directly in the runtime context instead of having to parse them or query the API.

Hope this helps, 😥
Isi