[Databricks Assets Bundles] no deployment state

[jorperort]

Good morning,

I'm trying to run:

databricks bundle run --debug -t dev integration_tests_job

My bundle looks:

bundle:
  name: x

include:
  - ./resources/*.yml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: x
    run_as:
      user_name: x

  prod:
    mode: production
    workspace:
      host: x
    run_as:
      user_name: x

resources:
  jobs:
    integration_tests_job:
      name: integration_tests_job

      email_notifications:
        on_failure:
          - x

      tasks:
        - task_key: notebook_task
          job_cluster_key: job_cluster
          notebook_task:
            notebook_path: ../tests/integration/main.py

      job_clusters:
        - job_cluster_key: job_cluster
          existing_cluster_id: x

And I'm getting this error:

10:56:28 ERROR Error: no deployment state. Did you forget to run 'databricks bundle deploy'? pid=265687 mutator=seq mutator=terraform.Load
10:56:28 ERROR Error: no deployment state. Did you forget to run 'databricks bundle deploy'? pid=265687 mutator=seq
Error: no deployment state. Did you forget to run 'databricks bundle deploy'?
10:56:28 ERROR failed execution pid=265687 exit_code=1 error="no deployment state. Did you forget to run 'databricks bundle deploy'?"

 I observe that the deployment seems to be carried out correctly, creating /Workspace/Users/x/.bundle/x/dev/state/terraform.tfstate

{
  "version": 4,
  "terraform_version": "1.5.5",
  "serial": 1,
  "lineage": "x",
  "outputs": {},
  "resources": [],
  "check_results": null
}

Could you help me with the error?

Jordi

[Kaniz]

Hi @jorperort, 

The error message you’re seeing, “no deployment state. Did you forget to run ‘databricks bundle deploy’?”, indicates that the deployment state is missing.

Here are some steps you can take to resolve this issue:

1. Verify Deployment:

   • Double-check that you’ve actually deployed your bundle using the databricks bundle deploy command. If you haven’t deployed it yet, make sure to do so before running the databricks bundle run command.
   • If you’ve already deployed it, ensure that the deployment was successful and that the necessary resources are in place.

2. Check Environment Variables:

   • Confirm that your environment variables are correctly set up. Specifically, check the following:
     • DATABRICKS_HOST: This should be the URL of your Databricks workspace.
     • DATABRICKS_TOKEN: Make sure this token is valid and has the necessary permissions.
   • If any of these variables are missing or incorrect, update them accordingly.

3. Bundle Configuration:

   • Review your bundle configuration file (x.yml). Ensure that the paths and settings are correctly specified.
   • Pay attention to the targets section, especially the dev mode. Make sure it points to the correct workspace and user.

4. Check Deployment State:

   • The deployment state is stored in the .bundle/x/dev/state/terraform.tfstate file. Verify that this file exists and contains the necessary information.
   • If it’s missing or incomplete, consider redeploying your bundle.

5. Terraform Version:

   • The version of Terraform you’re using matters. In your case, it’s version 1.5.5. Ensure that this version is compatible with your Databricks setup.

6. Integration Tests:

   • Since you’re running integration tests, ensure that the necessary resources (clusters, notebooks, etc.) are available in your Databricks workspace.
   • Check if there are any issues with the integration test notebook (../tests/integration/main.py).

Remember to follow the CI/CD best practices for Databricks bundles. If you’re using Azure DevOps, you can integrate it with Databricks Repos API for a seamless CI/CD pi...¹. If you encounter any specific errors during these steps, feel free to share them, and we can dive deeper into troubleshooting!

¹: Implementing CI/CD on Databricks Using Databricks Notebooks and Azure DevOps

[jorperort]

Hi @Kaniz , I'm going to review what you're saying in detail. Anyway, let me comment in case you have any ideas. I'm going point by point.

1. I have executed the deploy command and the files are deployed in the correct path. I also see the Terraform state file locally and in the path I mentioned earlier (I have also put the content of the file). I see the folders of artifacts, files, and states in the Databricks path.

2. I'm going to review this thoroughly. I have the Databricks CLI authenticated, and the deploy is done correctly. What I think I don't have configured are the environment variables (The host is retrieved from the DAB .yaml file in the workspace, if I'm not mistaken, right?).

3. My configuration seems correct, but I'll review it thoroughly just in case.

4. The status is what I mentioned before, apparently everything is correct.

5. I also need to check this because my version of Databricks is 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12). Where can I find which version of Databricks is compatible with version 1.5.5.?

6. Thank you very much for the information. Let's see if I can manage to execute the run command and make some changes recommended by the URL.

[jorperort]

Hello @Kaniz, I have been running tests and I can't get it to work. The problem arises when running the integration test job. As a temporary measure to avoid getting stuck, I've used the Databricks CLI (jobs) to create and execute the job, and it works correctly. All these tests are being conducted from my local machine with user authentication. Once it's working, I'll use a service principal from Azure DevOps pipelines.

I want to inform you that I have authenticated the CLI as follows, and I have configured the environment variables in this way:

; The profile defined in the DEFAULT section is to be used as a fallback when no profile is explicitly specified.
[DEFAULT]

[adb-x]
host      = x
auth_type = databricks-cli

export DATABRICKS_HOST="x"
export DATABRICKS_TOKEN="x"

I have conducted tests by modifying fields in the YAML, for instance, adding:

permissions:
  - level: CAN_RUN
    user_name: x

run_as:
  - user_name: x

I can't seem to make it work. The deployment is successful, do you have any idea what might be affecting the 'run' command to return 'ERROR Error: no deployment state. Did you forget to run 'databricks bundle deploy'? pid=265687 mutator=seq mutator=terraform.Load'?

Jordi