Scheduling a Complete Python Project in Databricks

[naineel]

Hi everyone,

I have a simple Python project with the following structure:

root/  
│── src/  
│   ├── package_name/  
│   │   ├── __init__.py  
│   │   ├── main.py  
│   │   ├── submodules1/  
│   │   │   ├── __init__.py  
│   │   │   ├── base1.py  
│   │   ├── submodules2/  
│   │   │   ├── __init__.py  
│   │   │   ├── base2.py  
│── pyproject.toml  

My primary concern is that while I can schedule a notebook or a single Python file in Databricks, I am unable to schedule an entire project that depends on main.py. The project functions as a CLI application and requires arguments to run.

I need guidance on how to properly schedule this complete Python project in Databricks. Any suggestions or best practices would be greatly appreciated!

Thanks in advance.

[ashraf1395]

Hi there @naineel , one approach can be you can convert your project into a whl file
and then create a python whl task for it and schedule

https://docs.databricks.com/aws/en/jobs/python-wheel