Passing UNIX-based parameter to a task

Garrus990 — Tue, 15 Oct 2024 20:13:53 GMT

Hey,

I would like to pass to a task a parameter that is based on a UNIX function. Concretely, I would like to specify dates - dynamically calculated with respect to the date of running my job. I wanted to it like that:

["--period-start", "$(date -d '-7 days' +'%Y-%m-%d')", "--period-end", "$(date -d '-1 days' +'%Y-%m-%d')"]

It doesn't seem to work, though. Is there any way of achieving it?

Re: Passing UNIX-based parameter to a task

NandiniN — Fri, 01 Nov 2024 05:31:37 GMT

Hi @Garrus990 ,

To pass a parameter to a task that is based on a UNIX function, you can use the Databricks Jobs API to dynamically calculate dates with respect to the date of running your job.

Use a Notebook to Calculate Dates: Create a notebook that calculates the required dates using Python or Scala. For example, in Python:

%py 
from datetime import datetime, timedelta

 period_start = (datetime.now() - timedelta(days=7)).strftime('%Y-%m-%d')
 period_end = (datetime.now() - timedelta(days=1)).strftime('%Y-%m-%d')

 dbutils.notebook.exit(f'{{"period_start": "{period_start}", "period_end": "{period_end}"}}')

Run the Notebook as a Job: Configure this notebook as a job in Databricks. This job will calculate the dates and return them as a JSON string.
Use the Output in Subsequent Tasks: In your main job, use the output of the first notebook as input parameters for subsequent tasks. You can achieve this by chaining jobs and passing the output of one job to another.

Here is an example of how you can set up the job configuration:

Job 1: Calculate Dates
- Notebook: calculate_dates
- Output: {"period_start": "2024-10-25", "period_end": "2024-10-31"}
Job 2: Main Job
- Notebook: main_job
- Parameters: --period-start {{job1_output.period_start}} --period-end {{job1_output.period_end}}

Thanks!

topic Re: Passing UNIX-based parameter to a task in Data Engineering

Passing UNIX-based parameter to a task

Re: Passing UNIX-based parameter to a task