cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

gbrueckl
by Contributor II
  • 12022 Views
  • 16 replies
  • 3 kudos

Resolved! Setup Git Integration via REST API

We are currently setting up CI/CD for our Databricks workspace using Databricks Repos following the approach described in the offical docs: https://docs.databricks.com/repos.html#best-practices-for-integrating-databricks-repos-with-cicd-workflowsObvi...

  • 12022 Views
  • 16 replies
  • 3 kudos
Latest Reply
New1
New Contributor II
  • 3 kudos

Hi, how can i trigger a job externally using Github actions?

  • 3 kudos
15 More Replies
Sarvagna_Mahaka
by New Contributor III
  • 3877 Views
  • 3 replies
  • 1 kudos

Resolved! Unable to clone GitLab Enterprise Edition repo in Databricks

Below are the steps that I followed. I still get an error message.Create a repo in gitlab enterprise editionIn GitLab, create a personal access token that allows access to your repositories ( with read_repository and write_repository permissions)Save...

error
  • 3877 Views
  • 3 replies
  • 1 kudos
Latest Reply
User16725394280
Contributor II
  • 1 kudos

Hi @Sarvagna Mahakali​  the repository which you are trying to add might be behind the VPN, our service cannot access it since it has no access to the VPN network.You may need the Enterprise Git / VPC to connect to the repository.Kindly check and let...

  • 1 kudos
2 More Replies
alejandrofm
by Valued Contributor
  • 2486 Views
  • 2 replies
  • 0 kudos

Resolved! Can't enable CLI 2.1 on CI

Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?Error:Resetting Databricks Job with job_id 1036...WARN: Your CLI is configured to use...

  • 2486 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Alejandro Martinez​ , To set up and use the Databricks jobs CLI (and the job runs CLI) to call the Jobs REST API 2.1, Update the CLI to version 0.16.0 or above.Run pip install databricks-cli --upgrade using the appropriate version of pip for your...

  • 0 kudos
1 More Replies
maranBH
by New Contributor III
  • 1852 Views
  • 4 replies
  • 1 kudos

Resolved! Trained model artifact, CI/CD and Databricks without MLFlow.

Hi all,We are constructing our CI/CD pipelines with the Repos feature following this guide:https://databricks.com/blog/2021/09/20/part-1-implementing-ci-cd-on-databricks-using-databricks-notebooks-and-azure-devops.htmlI'm trying to implement my pipes...

  • 1852 Views
  • 4 replies
  • 1 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

So you are managing your models with MLflow, and want to include them in a git repository?You can do that in a CI/CD process; it would run the mlflow CLI to copy the model you want (e.g. model:/my_model/production) to a git checkout and then commit i...

  • 1 kudos
3 More Replies
MadelynM
by Contributor II
  • 919 Views
  • 0 replies
  • 1 kudos

vimeo.com

Repos let you use Git functionality such as cloning a remote repo, managing branches, pushing and pulling changes and visually comparing differences upon commit. Here's a quick video (3:56) on setting up a repo for Databricks on AWS. Pre-reqs: Git in...

  • 919 Views
  • 0 replies
  • 1 kudos
amichel
by New Contributor III
  • 4949 Views
  • 3 replies
  • 2 kudos

Resolved! Is there a way to refresh tokens issued on behalf of service principal?

I want to be able to refresh tokens generated on behalf of a service principal via Token Management API, just like with any other service where OAuth is used and refresh token endpoint is available. Allowing indefinite or very long expiration for acc...

  • 4949 Views
  • 3 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Refresh option would be useful.In Azure you could use Azure automation to make "refresh" script: delete if still existscreate token via: "databricks tokens create" put it to Azure Key Vault with expiration data

  • 2 kudos
2 More Replies
MadelynM
by Contributor II
  • 2379 Views
  • 2 replies
  • 1 kudos

2021-08-Best-Practices-for-Your-Data-Architecture-v3-OG-1200x628

Thanks to everyone who joined the Best Practices for Your Data Architecture session on Getting Workloads to Production using CI/CD. You can access the on-demand session recording here, and the code in the Databricks Labs CI/CD Templates Repo. Posted ...

  • 2379 Views
  • 2 replies
  • 1 kudos
Latest Reply
MadelynM
Contributor II
  • 1 kudos

Here's the embedded links list!Jobs scheduling and orchestrationBuilt-in job scheduling: https://docs.databricks.com/jobs.html#schedule-a-job Periodic scheduling of the jobsExecute notebook / jar / Python script / Spark-submitMultitask JobsExecute no...

  • 1 kudos
1 More Replies
RantoB
by Valued Contributor
  • 1984 Views
  • 2 replies
  • 4 kudos

Resolved! Import a notebook in a Release Pipeline with a Python script

Hi, I would like to import a python file to Databricks with a Azure DevOps Release Pipeline.Within the pipeline I execute a python script which contains this code :import sys import os import base64 import requests   dbw_url = sys.argv[1] # https://a...

  • 1984 Views
  • 2 replies
  • 4 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

Recently I wrote about alternative way to export/import notebooks in pthon https://community.databricks.com/s/question/0D53f00001TgT52CAF/import-notebook-with-python-script-using-api This way you will get more readable error/message (often it is rela...

  • 4 kudos
1 More Replies
Abhendu
by New Contributor II
  • 1420 Views
  • 3 replies
  • 0 kudos

Resolved! CICD Databricks

Hi TeamI was wondering if there is a document or step by step process to promote code in CICD across various environments of code repository (GIT/GITHUB/BitBucket/Gitlab) with DBx support? [Without involving code repository merging capability of the ...

  • 1420 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Please refer this related thread on CICD in Databricks https://community.databricks.com/s/question/0D53f00001GHVhMCAX/what-are-some-best-practices-for-cicd

  • 0 kudos
2 More Replies
Labels