05-02-2024 02:08 AM - edited 05-02-2024 02:11 AM
Good morning,
I'm trying to run:
databricks bundle run --debug -t dev integration_tests_job
My bundle looks:
bundle:
name: x
include:
- ./resources/*.yml
targets:
dev:
mode: development
default: true
workspace:
host: x
run_as:
user_name: x
prod:
mode: production
workspace:
host: x
run_as:
user_name: x
resources:
jobs:
integration_tests_job:
name: integration_tests_job
email_notifications:
on_failure:
- x
tasks:
- task_key: notebook_task
job_cluster_key: job_cluster
notebook_task:
notebook_path: ../tests/integration/main.py
job_clusters:
- job_cluster_key: job_cluster
existing_cluster_id: x
And I'm getting this error:
10:56:28 ERROR Error: no deployment state. Did you forget to run 'databricks bundle deploy'? pid=265687 mutator=seq mutator=terraform.Load
10:56:28 ERROR Error: no deployment state. Did you forget to run 'databricks bundle deploy'? pid=265687 mutator=seq
Error: no deployment state. Did you forget to run 'databricks bundle deploy'?
10:56:28 ERROR failed execution pid=265687 exit_code=1 error="no deployment state. Did you forget to run 'databricks bundle deploy'?"
I observe that the deployment seems to be carried out correctly, creating /Workspace/Users/x/.bundle/x/dev/state/terraform.tfstate
{
"version": 4,
"terraform_version": "1.5.5",
"serial": 1,
"lineage": "x",
"outputs": {},
"resources": [],
"check_results": null
}
Could you help me with the error?
Jordi
05-03-2024 02:31 AM
Verify Deployment:
databricks bundle deploy
command. If you haven’t deployed it yet, make sure to do so before running the databricks bundle run
command.Check Environment Variables:
Bundle Configuration:
x.yml
). Ensure that the paths and settings are correctly specified.targets
section, especially the dev
mode. Make sure it points to the correct workspace and user.Check Deployment State:
.bundle/x/dev/state/terraform.tfstate
file. Verify that this file exists and contains the necessary information.Terraform Version:
Integration Tests:
../tests/integration/main.py
).Remember to follow the CI/CD best practices for Databricks bundles. If you’re using Azure DevOps, you can integrate it with Databricks Repos API for a seamless CI/CD pi...1. If you encounter any specific errors during these steps, feel free to share them, and we can dive deeper into troubleshooting!
1: Implementing CI/CD on Databricks Using Databricks Notebooks and Azure DevOps
05-03-2024 03:56 AM
Hi @Kaniz , I'm going to review what you're saying in detail. Anyway, let me comment in case you have any ideas. I'm going point by point.
I have executed the deploy command and the files are deployed in the correct path. I also see the Terraform state file locally and in the path I mentioned earlier (I have also put the content of the file). I see the folders of artifacts, files, and states in the Databricks path.
I'm going to review this thoroughly. I have the Databricks CLI authenticated, and the deploy is done correctly. What I think I don't have configured are the environment variables (The host is retrieved from the DAB .yaml file in the workspace, if I'm not mistaken, right?).
My configuration seems correct, but I'll review it thoroughly just in case.
The status is what I mentioned before, apparently everything is correct.
I also need to check this because my version of Databricks is 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12). Where can I find which version of Databricks is compatible with version 1.5.5.?
Thank you very much for the information. Let's see if I can manage to execute the run command and make some changes recommended by the URL.
05-06-2024 01:31 AM - edited 05-06-2024 01:33 AM
Hello @Kaniz, I have been running tests and I can't get it to work. The problem arises when running the integration test job. As a temporary measure to avoid getting stuck, I've used the Databricks CLI (jobs) to create and execute the job, and it works correctly. All these tests are being conducted from my local machine with user authentication. Once it's working, I'll use a service principal from Azure DevOps pipelines.
I want to inform you that I have authenticated the CLI as follows, and I have configured the environment variables in this way:
; The profile defined in the DEFAULT section is to be used as a fallback when no profile is explicitly specified.
[DEFAULT]
[adb-x]
host = x
auth_type = databricks-cli
export DATABRICKS_HOST="x"
export DATABRICKS_TOKEN="x"
I have conducted tests by modifying fields in the YAML, for instance, adding:
permissions:
- level: CAN_RUN
user_name: x
run_as:
- user_name: x
I can't seem to make it work. The deployment is successful, do you have any idea what might be affecting the 'run' command to return 'ERROR Error: no deployment state. Did you forget to run 'databricks bundle deploy'? pid=265687 mutator=seq mutator=terraform.Load'?
Jordi
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!