Hey @Forssen ,
My advice:
Using .py files and .whl packages is generally more secure and scalable, especially when working in a team. One of the key advantages is that code reviews and version control are much more efficient with .py files, as changes can be properly tracked via pull requests.
While notebooks can have permissions set for reading and version control, they are often harder to manage in collaborative environments. A common issue is that people forget to remove unnecessary display() statements or collect(), which makes reviewing and debugging easier in a notebook but is considered bad practice in production. In addition, a single "," inserted in the notebook accidentally can make your production job fail.
Advantages of .py and .whl over notebooks:
•Better version control & code reviews (easier to track changes and enforce coding standards).
•Better modularization & reusability (separating logic into reusable components).
•Easier CI/CD integration (you can automate testing, packaging, and deployment).
•More structured and maintainable codebase (better organization and scalability).
Disadvantages:
•Harder debugging compared to notebooks (notebooks allow quick testing and visualization).
•Steeper learning curve for new users who are used to interactive workflows.
Given your current setup, where you use notebooks only as orchestrators and keep your logic in .py modules, you already have a good balance. The next step could be fully transitioning orchestration to workflows (like Airflow or Databricks Jobs) and packaging your code into .whl files for better maintainability.
🙂
