- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2025 10:29 AM
Hello,
Thank you for your amazing response! You provided all the information I needed at this moment, and I truly appreciate it.
However, I have a doubt regarding the following example code:
(Old) file.py - Based on Airflow DAGs:
with (...):
@task
def task_1():
...
process_email()
...
@task
def task_2():
...
(New) file.py - Databricks Workflow:
Here, all task definitions are consolidated into a single notebook file, with callable functions that execute the required tasks using the base_parameters passed for example calling function: "task_1", in the other type of task, function: "task_2".
But I think I made a mistake in the settings. Instead of properly splitting the old file.py into individual tasks (where each task corresponds to a separate notebook file), we replicated the entire structure for each task in one notebook.
Now, regarding the process_email method inside task_1:
- Should I create a separate task for the process_email functionality?
- Currently, I placed the process_email function inside the src folder and imported it using a wheel package. Is this the correct approach?
Additionally, could you clarify which parts of the code are most suitable for inclusion in a wheel package file?
Thank you again for your help!