Hello Lebrown:

Here are the steps I managed to deploy job and pipelines to Databricks using DABs with example (Using free edition):

1. Install the Databricks CLI (latest version). On Windows, run:
winget search databricks
winget install Databricks.DatabricksCLI

2. Authenticate the CLI:
Run: databricks configure

 - Enter your Databricks workspace URL (e.g. https://<your-workspace>.databricks.com)

 - Go to User Settings → Developer → Manage → Generate New Token

 - Paste the token when prompted

3. Initialize the project:
databricks bundle init default-python

4. Install required Python libraries:
pip install -r requirements-dev.txt

5. In your job or pipeline YAML files:

 - Use serverless compute instead of defining job clusters
 - For pipelines, add serverless: true under the pipeline section
 - Use %pip magic command to install libraries instead of whl.

6. Deploy the bundle:
databricks bundle deploy -t dev


let me know if you need any other help.

Best, Ilir