jeremy98
Honored Contributor

Hello,

Thank you for your amazing response! You provided all the information I needed at this moment, and I truly appreciate it.

However, I have a doubt regarding the following example code:

(Old) file.py - Based on Airflow DAGs:

 

 

with (...):  
    @task  
    def task_1():  
        ...  
        process_email()  
        ...  

    @task  
    def task_2():  
        ...  

 

 

(New) file.py - Databricks Workflow:
Here, all task definitions are consolidated into a single notebook file, with callable functions that execute the required tasks using the base_parameters passed for example calling function: "task_1", in the other type of task, function: "task_2".

But I think I made a mistake in the settings. Instead of properly splitting the old file.py into individual tasks (where each task corresponds to a separate notebook file), we replicated the entire structure for each task in one notebook.

Now, regarding the process_email method inside task_1:

  • Should I create a separate task for the process_email functionality?
  • Currently, I placed the process_email function inside the src folder and imported it using a wheel package. Is this the correct approach?

Additionally, could you clarify which parts of the code are most suitable for inclusion in a wheel package file?

Thank you again for your help!