cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

joao_vnb
by New Contributor III
  • 20447 Views
  • 7 replies
  • 11 kudos

Resolved! Automate the Databricks workflow deployment

Hi everyone,Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?

  • 20447 Views
  • 7 replies
  • 11 kudos
Latest Reply
asingamaneni
New Contributor II
  • 11 kudos

Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflowYou can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic w...

  • 11 kudos
6 More Replies
francly
by New Contributor II
  • 1729 Views
  • 5 replies
  • 3 kudos

Resolved! terraform create multiple db user

Hi, follow the example to create one user. It's working however I want to create multiple users, I have tried many ways but still cannot get it work, please share some idea.https://registry.terraform.io/providers/databricks/databricks/latest/docs/res...

  • 1729 Views
  • 5 replies
  • 3 kudos
Latest Reply
Natlab
New Contributor II
  • 3 kudos

What if I want to give User Name along with the email ID?I used below code but its not helping(code is not failing, but not adding user name)It seems this code line: "display_name = each.key" is not working. Pls suggest.  terraform {required_provider...

  • 3 kudos
4 More Replies
Suman
by New Contributor III
  • 598 Views
  • 2 replies
  • 2 kudos

Can Lint be implemented in multilingual databricks notebook. ?

I want to implement lint in my project as part of CICD. But as my notebooks has a mix of python and scala . Can Lint be implemented there ?

  • 598 Views
  • 2 replies
  • 2 kudos
Latest Reply
Suman
New Contributor III
  • 2 kudos

@Vidula Khanna​ I didn't hear from anyone . I am looking for this answer to implement in my databricks code of my project.Can you please assist.

  • 2 kudos
1 More Replies
DJey
by New Contributor III
  • 3008 Views
  • 4 replies
  • 3 kudos

Databricks CI/CD Azure DevOps

Hi All. I have a scenario where there are few .sql scripts present in my repo. Is there any way we can execute those SQLs on Databricks via Azure DevOps CI/CD pipeline?Please help.

  • 3008 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Divyansh Jain​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...

  • 3 kudos
3 More Replies
SK21
by New Contributor II
  • 946 Views
  • 3 replies
  • 1 kudos

CICD for Jobs @ WorkFlows

I had created Jobs to trigger the respective notebooks in Databricks Workflow.Now I need to move them to further environments.Would you please help me with an CICD process to promote jobs to further environments.

  • 946 Views
  • 3 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Please use jobs API 2.1 You can get job and save JSON with that jobs to git.In git then set variables defining databricks workspaces (URL and token) and after push define that API call is triggered with your json stored in git.

  • 1 kudos
2 More Replies
labromb
by Contributor
  • 2226 Views
  • 7 replies
  • 6 kudos

Databricks Jobs and CICD

Hi, We currently leverage Azure DevOps to source control our notebooks and use CICD to publish the notebooks to different environments and this works very well. We do not have the same functionality available for Databricks jobs (the ability to sourc...

  • 2226 Views
  • 7 replies
  • 6 kudos
Latest Reply
JRT5933
New Contributor III
  • 6 kudos

My team is currently looking at establishing REPO(s) for source control to start. I know I've seen some documentation for when a MERGE is completed to auto update the main branch in DBX remote repo. Does annyone have a template and/or best practices ...

  • 6 kudos
6 More Replies
cmilligan
by Contributor II
  • 551 Views
  • 1 replies
  • 1 kudos

Resolved! Database CICD Pipelines

My team has a shared codebase and we are running into issues as we migrate to Databricks when two people are doing development on connected sections of our codebase.For example if I add a column to a table for changes on my branch, other members on m...

  • 551 Views
  • 1 replies
  • 1 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 1 kudos

@Coleman Milligan​ It's really hard to create something like this without basic knowledge about how CICD should work or even Terraform.You can start here, to understand some basics.https://servian.dev/how-to-hardening-azure-databricks-using-terraform...

  • 1 kudos
Vsleg
by Contributor
  • 1391 Views
  • 2 replies
  • 1 kudos

Resolved! Deploying Databricks Workflows and Delta Live Table pipelines across workspaces

Hello,I was wondering if there is a way to deploy Databricks Workflows and Delta Live Table pipelines across Workspaces (DEV/UAT/PROD) using Azure DevOps.

  • 1391 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Yes, for sure, using Rest API Calls to https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-api-guide.htmlYou can create DLT manually from GUI and take JSON representative of it, tweak it (so it uses your env variables, for examp...

  • 1 kudos
1 More Replies
isaac_gritz
by Valued Contributor II
  • 7363 Views
  • 6 replies
  • 6 kudos

Local Development on Databricks

How to Develop Locally on Databricks with your Favorite IDEdbx is a Databricks Labs project that allows you to develop code locally and then submit against Databricks interactive and job compute clusters from your favorite local IDE (AWS | Azure | GC...

  • 7363 Views
  • 6 replies
  • 6 kudos
Latest Reply
Jfoxyyc
Valued Contributor
  • 6 kudos

I'm actually not a fan of dbx. I prefer the AWS Glue interactive sessions way of using the IDE. It's exactly like the web notebook experience. I can see the reason why dbx exists, but I'd still like to use a regular notebook experience in my IDE.

  • 6 kudos
5 More Replies
Anonymous
by Not applicable
  • 4636 Views
  • 14 replies
  • 8 kudos

Resolved! What are some best practices for CICD?

A number of people have questions on using Databricks in a productionalized environment. What are the best practices to enable CICD automation?

  • 4636 Views
  • 14 replies
  • 8 kudos
Latest Reply
xiangzhu
Contributor
  • 8 kudos

https://github.com/databrickslabs/cicd-templates is legacy now, the updated one here: dbx.you need to walkthrough the doc carefully, there're many informations inside, and BTW `dbx init` could create a template.

  • 8 kudos
13 More Replies
Pat
by Honored Contributor III
  • 1152 Views
  • 4 replies
  • 19 kudos

Resolved! UC - Service Principal/Terraform

Hi,do you know if there is a way to create Unity Catalog metastore using Service Principal?Here I can see that for creating account-level resources we need to provide a user and password (https://registry.terraform.io/providers/databricks/databricks/...

  • 1152 Views
  • 4 replies
  • 19 kudos
Latest Reply
Pat
Honored Contributor III
  • 19 kudos

This is supported right now in the Azure, but not yet in AWS, but there is plan for AWS support as well.

  • 19 kudos
3 More Replies
RantoB
by Valued Contributor
  • 4021 Views
  • 10 replies
  • 8 kudos

Resolved! Import notebook with python script using API

Hi,I would like to import a python notebook to my databricks workspace from my local machine using a python script.I manages to create the folder but then I have a status code 400 when I try to import a file :create_folder = requests.post( '{}/api/...

  • 4021 Views
  • 10 replies
  • 8 kudos
Latest Reply
RantoB
Valued Contributor
  • 8 kudos

Hi, Thanks for your answer.Actually both your code and mine are working. However, I cannot write in the directory Repos which is reserved (but I can create subdirectories...)Thanks to your code I got an error message which helped me to understand. Wi...

  • 8 kudos
9 More Replies
isaac_gritz
by Valued Contributor II
  • 1235 Views
  • 1 replies
  • 5 kudos

Resolved! CI/CD Best Practices

Best Practices for CI/CD on DatabricksFor CI/CD and software engineering best practices with Databricks notebooks we recommend checking out this best practices guide (AWS, Azure, GCP).For CI/CD and local development using an IDE, we recommend dbx, a ...

  • 1235 Views
  • 1 replies
  • 5 kudos
Latest Reply
Kaniz
Community Manager
  • 5 kudos

Thank you, @Isaac Gritz​ , for sharing such a fantastic post!

  • 5 kudos
mehdi1
by New Contributor III
  • 5209 Views
  • 9 replies
  • 12 kudos

Resolved! How to programmatically create a widget?

I know the dbutils.widget.text to create a widget in a notebook. So for me the workflow 1. Having a notebook2. Use dbutils.widget.text (or other type of widgets) once in a notebook cell to create a widget3. Remove the cell containing dbutils.widget...

  • 5209 Views
  • 9 replies
  • 12 kudos
Latest Reply
RachelGomez123
New Contributor II
  • 12 kudos

@Mehdi BEN ABDESSELEM​ , Steps for Creating a Basic WidgetStep 1: Create a New ProjectTo create a new project in Android Studio, please refer to How to Create/Start a New Project in Android Studio. We are implementing it for both Java and Kotlin lang...

  • 12 kudos
8 More Replies
Labels