โ11-05-2024 01:59 AM
I need to access the date of a given job running as a non-notebook Python job (spark_python_task). I want to pass a value from the cli when running it and being available to access the value in the script
I tried the approaches in the attached image when running
bundle run my_job --params run_date=20240101
โ11-05-2024 05:48 AM
Job parameters are automatically pushed down as key-value parameters to all tasks that accept key-value parameters, which include the following task types:
Notebook
Python wheel (only when configured with keyword arguments)
SQL query, legacy dashboard, or file
Run Job
โ11-05-2024 07:48 AM
Hi, thank you for your response. I have a few follow-up questions to clarify best practices when it comes to passing parameters with Python files:
Thank you for your help with this!
โ11-05-2024 09:08 AM
I come with 1 more question. To clarify, my previous questions were focused on jobs that are triggered manually, without scheduling.
For scheduled jobsโparticularly those using Python script tasksโhow can I configure the `job.yml` resource and the Python script to dynamically retrieve `{{job.start_time.iso_date}}` at runtime?
Thanks again!
โ11-05-2024 09:40 AM
Can you confirm if this solution applies to you https://community.databricks.com/t5/data-engineering/retrieve-job-level-parameters-in-spark-python-t... ?
โ11-05-2024 09:48 AM
So, I think if I change the spark_python_task to a notebook_task but I keep the file as python file instead of notebook is ok. Now I can use the data bricks widgets easily and retrieve those parameters and I also put to version control python files instead of notebook
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group