Scheduling a Complete Python Project in Databricks

naineel — Tue, 01 Apr 2025 03:37:29 GMT

Hi everyone,

I have a simple Python project with the following structure:

root/  
│── src/  
│   ├── package_name/  
│   │   ├── __init__.py  
│   │   ├── main.py  
│   │   ├── submodules1/  
│   │   │   ├── __init__.py  
│   │   │   ├── base1.py  
│   │   ├── submodules2/  
│   │   │   ├── __init__.py  
│   │   │   ├── base2.py  
│── pyproject.toml

My primary concern is that while I can schedule a notebook or a single Python file in Databricks, I am unable to schedule an entire project that depends on main.py. The project functions as a CLI application and requires arguments to run.

I need guidance on how to properly schedule this complete Python project in Databricks. Any suggestions or best practices would be greatly appreciated!

Thanks in advance.

Re: Scheduling a Complete Python Project in Databricks

ashraf1395 — Tue, 01 Apr 2025 04:33:48 GMT

Hi there @naineel , one approach can be you can convert your project into a whl file
and then create a python whl task for it and schedule

https://docs.databricks.com/aws/en/jobs/python-wheel

topic Scheduling a Complete Python Project in Databricks in Data Engineering

Scheduling a Complete Python Project in Databricks

Re: Scheduling a Complete Python Project in Databricks