marcuskw
Databricks Partner

Hi  
We have a Factory logic that looks something like this:

 

class Factory:
    def __init__(self, job_parameter: str): 
        self.job_parameter = job_parameter 
    
    def set_objects(self): 
        if self.job_parameter == "A": 
            from path.A import LogicClass 

        elif self.job_parameter == "B": 
            from path.B import LogicClass

        return LogicClass

 

The aim is to have a generic notebook that would then look like this:

 

from pyspark.dbutils import DBUtils 
from path.factory import Factory 

job_parameter = dbutils.widgets.get("job_parameter") 
LogicClass= Factory(job_parameter).set_objects()
LogicClass.run_business_logic()

 

When we want to use orchestration we are forced to create multiple jobs:

  • Job "A" which runs the generic notebook with "job_parameter" = "A" with a schedule
  • Job "B" which runs the generic notebook with "job_parameter" = "B" with a schedule

A solution where we only have 1 job would help us here where we have:

  • Job "Run generic notebook"
    • Schedule 1 with "job_parameter" = "A"
    • Schedule 2 with "job_parameter" = "B"