cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How can retrieve backfill run parameter in Python?

flourishingsing
New Contributor III

flourishingsing_0-1779284296139.png

I'm trying to run backfill with the following parameter. How can I access this in the Python script?
Do I need to change anything in the yml?

I usually set task parameters the following way:

flourishingsing_1-1779284438804.png

These are then parsed using argparse Python module.

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

flourishingsing
New Contributor III

Found the following solution:
Add job level parameters:

parameters:
  - name: run_timestamp
    default: "some_default_value"

 

Reference in task level parameters:

tasks:
  - task_key: my_task
    spark_python_task:
      python_file: ../../script.py
      parameters:
        - --run-timestamp
        - "{{job.parameters.run_timestamp}}"

 

Deploy to Databricks and override default value of job level parameter when triggering the backfill runs.

View solution in original post

1 REPLY 1

flourishingsing
New Contributor III

Found the following solution:
Add job level parameters:

parameters:
  - name: run_timestamp
    default: "some_default_value"

 

Reference in task level parameters:

tasks:
  - task_key: my_task
    spark_python_task:
      python_file: ../../script.py
      parameters:
        - --run-timestamp
        - "{{job.parameters.run_timestamp}}"

 

Deploy to Databricks and override default value of job level parameter when triggering the backfill runs.