Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2024 06:30 AM - edited 01-12-2024 06:40 AM
Hi
We have a Factory logic that looks something like this:
class Factory:
def __init__(self, job_parameter: str):
self.job_parameter = job_parameter
def set_objects(self):
if self.job_parameter == "A":
from path.A import LogicClass
elif self.job_parameter == "B":
from path.B import LogicClass
return LogicClass
The aim is to have a generic notebook that would then look like this:
from pyspark.dbutils import DBUtils
from path.factory import Factory
job_parameter = dbutils.widgets.get("job_parameter")
LogicClass= Factory(job_parameter).set_objects()
LogicClass.run_business_logic()
When we want to use orchestration we are forced to create multiple jobs:
- Job "A" which runs the generic notebook with "job_parameter" = "A" with a schedule
- Job "B" which runs the generic notebook with "job_parameter" = "B" with a schedule
A solution where we only have 1 job would help us here where we have:
- Job "Run generic notebook"
- Schedule 1 with "job_parameter" = "A"
- Schedule 2 with "job_parameter" = "B"