- 2831 Views
- 2 replies
- 0 kudos
Can't enable CLI 2.1 on CI
Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?Error:Resetting Databricks Job with job_id 1036...WARN: Your CLI is configured to use...
- 2831 Views
- 2 replies
- 0 kudos
- 10348 Views
- 15 replies
- 8 kudos
Resolved! What are some best practices for CICD?
A number of people have questions on using Databricks in a productionalized environment. What are the best practices to enable CICD automation?
- 10348 Views
- 15 replies
- 8 kudos
- 8 kudos
Any leads/posts for Databricks CI/CD integration with Bitbucket pipeline. I am facing the below error while I creation my CICD pipeline pipelines:branches:master:- step:name: Deploy Databricks Changesimage: docker:19.03.12services:- dockerscript:# U...
- 8 kudos
- 48781 Views
- 7 replies
- 11 kudos
Resolved! Automate the Databricks workflow deployment
Hi everyone,Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?
- 48781 Views
- 7 replies
- 11 kudos
- 11 kudos
Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflowYou can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic w...
- 11 kudos
- 3781 Views
- 5 replies
- 3 kudos
Resolved! terraform create multiple db user
Hi, follow the example to create one user. It's working however I want to create multiple users, I have tried many ways but still cannot get it work, please share some idea.https://registry.terraform.io/providers/databricks/databricks/latest/docs/res...
- 3781 Views
- 5 replies
- 3 kudos
- 3 kudos
What if I want to give User Name along with the email ID?I used below code but its not helping(code is not failing, but not adding user name)It seems this code line: "display_name = each.key" is not working. Pls suggest. terraform {required_provider...
- 3 kudos
- 1341 Views
- 2 replies
- 2 kudos
Can Lint be implemented in multilingual databricks notebook. ?
I want to implement lint in my project as part of CICD. But as my notebooks has a mix of python and scala . Can Lint be implemented there ?
- 1341 Views
- 2 replies
- 2 kudos
- 2 kudos
@Vidula Khanna​ I didn't hear from anyone . I am looking for this answer to implement in my databricks code of my project.Can you please assist.
- 2 kudos
- 9276 Views
- 4 replies
- 3 kudos
Databricks CI/CD Azure DevOps
Hi All. I have a scenario where there are few .sql scripts present in my repo. Is there any way we can execute those SQLs on Databricks via Azure DevOps CI/CD pipeline?Please help.
- 9276 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Divyansh Jain​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 3 kudos
- 678 Views
- 0 replies
- 0 kudos
Databricks CI/CD process is decoupled or coupled ?
In Databricks, CI/CD process is decoupled or coupled ?
- 678 Views
- 0 replies
- 0 kudos
- 1965 Views
- 3 replies
- 1 kudos
CICD for Jobs @ WorkFlows
I had created Jobs to trigger the respective notebooks in Databricks Workflow.Now I need to move them to further environments.Would you please help me with an CICD process to promote jobs to further environments.
- 1965 Views
- 3 replies
- 1 kudos
- 1 kudos
Please use jobs API 2.1 You can get job and save JSON with that jobs to git.In git then set variables defining databricks workspaces (URL and token) and after push define that API call is triggered with your json stored in git.
- 1 kudos
- 4582 Views
- 6 replies
- 7 kudos
Databricks Jobs and CICD
Hi, We currently leverage Azure DevOps to source control our notebooks and use CICD to publish the notebooks to different environments and this works very well. We do not have the same functionality available for Databricks jobs (the ability to sourc...
- 4582 Views
- 6 replies
- 7 kudos
- 7 kudos
My team is currently looking at establishing REPO(s) for source control to start. I know I've seen some documentation for when a MERGE is completed to auto update the main branch in DBX remote repo. Does annyone have a template and/or best practices ...
- 7 kudos
- 1365 Views
- 1 replies
- 1 kudos
Resolved! Database CICD Pipelines
My team has a shared codebase and we are running into issues as we migrate to Databricks when two people are doing development on connected sections of our codebase.For example if I add a column to a table for changes on my branch, other members on m...
- 1365 Views
- 1 replies
- 1 kudos
- 1 kudos
@Coleman Milligan​ It's really hard to create something like this without basic knowledge about how CICD should work or even Terraform.You can start here, to understand some basics.https://servian.dev/how-to-hardening-azure-databricks-using-terraform...
- 1 kudos
- 2373 Views
- 2 replies
- 1 kudos
Resolved! Deploying Databricks Workflows and Delta Live Table pipelines across workspaces
Hello,I was wondering if there is a way to deploy Databricks Workflows and Delta Live Table pipelines across Workspaces (DEV/UAT/PROD) using Azure DevOps.
- 2373 Views
- 2 replies
- 1 kudos
- 1 kudos
Yes, for sure, using Rest API Calls to https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-api-guide.htmlYou can create DLT manually from GUI and take JSON representative of it, tweak it (so it uses your env variables, for examp...
- 1 kudos
- 16579 Views
- 6 replies
- 6 kudos
Local Development on Databricks
How to Develop Locally on Databricks with your Favorite IDEdbx is a Databricks Labs project that allows you to develop code locally and then submit against Databricks interactive and job compute clusters from your favorite local IDE (AWS | Azure | GC...
- 16579 Views
- 6 replies
- 6 kudos
- 6 kudos
I'm actually not a fan of dbx. I prefer the AWS Glue interactive sessions way of using the IDE. It's exactly like the web notebook experience. I can see the reason why dbx exists, but I'd still like to use a regular notebook experience in my IDE.
- 6 kudos
- 2455 Views
- 3 replies
- 19 kudos
Resolved! UC - Service Principal/Terraform
Hi,do you know if there is a way to create Unity Catalog metastore using Service Principal?Here I can see that for creating account-level resources we need to provide a user and password (https://registry.terraform.io/providers/databricks/databricks/...
- 2455 Views
- 3 replies
- 19 kudos
- 19 kudos
This is supported right now in the Azure, but not yet in AWS, but there is plan for AWS support as well.
- 19 kudos
- 7884 Views
- 10 replies
- 8 kudos
Resolved! Import notebook with python script using API
Hi,I would like to import a python notebook to my databricks workspace from my local machine using a python script.I manages to create the folder but then I have a status code 400 when I try to import a file :create_folder = requests.post( '{}/api/...
- 7884 Views
- 10 replies
- 8 kudos
- 8 kudos
Hi, Thanks for your answer.Actually both your code and mine are working. However, I cannot write in the directory Repos which is reserved (but I can create subdirectories...)Thanks to your code I got an error message which helped me to understand. Wi...
- 8 kudos
- 2141 Views
- 0 replies
- 4 kudos
CI/CD Best Practices
Best Practices for CI/CD on DatabricksFor CI/CD and software engineering best practices with Databricks notebooks we recommend checking out this best practices guide (AWS, Azure, GCP).For CI/CD and local development using an IDE, we recommend dbx, a ...
- 2141 Views
- 0 replies
- 4 kudos