cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
mehdi1
by • New Contributor III
  • 9815 Views
  • 8 replies
  • 12 kudos

Resolved! How to programmatically create a widget?

I know the dbutils.widget.text to create a widget in a notebook. So for me the workflow 1. Having a notebook2. Use dbutils.widget.text (or other type of widgets) once in a notebook cell to create a widget3. Remove the cell containing dbutils.widget...

Data Engineering
Reply
  • 9815 Views
  • 8 replies
  • 12 kudos
View Replies
Latest Reply
RachelGomez123
New Contributor II
  • 12 kudos

@Mehdi BEN ABDESSELEM​ , Steps for Creating a Basic WidgetStep 1: Create a New ProjectTo create a new project in Android Studio, please refer to How to Create/Start a New Project in Android Studio. We are implementing it for both Java and Kotlin lang...

  • 12 kudos
7 More Replies
gbrueckl
by • Contributor II
  • 14035 Views
  • 9 replies
  • 2 kudos

Setup Git Integration via REST API

We are currently setting up CI/CD for our Databricks workspace using Databricks Repos following the approach described in the offical docs: https://docs.databricks.com/repos.html#best-practices-for-integrating-databricks-repos-with-cicd-workflowsObvi...

Data Engineering
Reply
  • 14035 Views
  • 9 replies
  • 2 kudos
View Replies
Latest Reply
New1
New Contributor II
  • 2 kudos

Hi, how can i trigger a job externally using Github actions?

  • 2 kudos
8 More Replies
Sarvagna_Mahaka
by • New Contributor III
  • 4746 Views
  • 3 replies
  • 1 kudos

Resolved! Unable to clone GitLab Enterprise Edition repo in Databricks

Below are the steps that I followed. I still get an error message.Create a repo in gitlab enterprise editionIn GitLab, create a personal access token that allows access to your repositories ( with read_repository and write_repository permissions)Save...

error
Data Engineering
Reply
  • 4746 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
User16725394280
Contributor II
  • 1 kudos

Hi @Sarvagna Mahakali​  the repository which you are trying to add might be behind the VPN, our service cannot access it since it has no access to the VPN network.You may need the Enterprise Git / VPC to connect to the repository.Kindly check and let...

  • 1 kudos
2 More Replies
maranBH
by • New Contributor III
  • 2209 Views
  • 3 replies
  • 1 kudos

Resolved! Trained model artifact, CI/CD and Databricks without MLFlow.

Hi all,We are constructing our CI/CD pipelines with the Repos feature following this guide:https://databricks.com/blog/2021/09/20/part-1-implementing-ci-cd-on-databricks-using-databricks-notebooks-and-azure-devops.htmlI'm trying to implement my pipes...

Data Engineering
Reply
  • 2209 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

So you are managing your models with MLflow, and want to include them in a git repository?You can do that in a CI/CD process; it would run the mlflow CLI to copy the model you want (e.g. model:/my_model/production) to a git checkout and then commit i...

  • 1 kudos
2 More Replies
MadelynM
by • Databricks Employee
  • 1077 Views
  • 0 replies
  • 1 kudos

vimeo.com

Repos let you use Git functionality such as cloning a remote repo, managing branches, pushing and pulling changes and visually comparing differences upon commit. Here's a quick video (3:56) on setting up a repo for Databricks on AWS. Pre-reqs: Git in...

Data Engineering
Reply
  • 1077 Views
  • 0 replies
  • 1 kudos
amichel
by • New Contributor III
  • 6150 Views
  • 3 replies
  • 2 kudos

Resolved! Is there a way to refresh tokens issued on behalf of service principal?

I want to be able to refresh tokens generated on behalf of a service principal via Token Management API, just like with any other service where OAuth is used and refresh token endpoint is available. Allowing indefinite or very long expiration for acc...

Data Engineering
Reply
  • 6150 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Refresh option would be useful.In Azure you could use Azure automation to make "refresh" script: delete if still existscreate token via: "databricks tokens create" put it to Azure Key Vault with expiration data

  • 2 kudos
2 More Replies
MadelynM
by • Databricks Employee
  • 2712 Views
  • 2 replies
  • 1 kudos

2021-08-Best-Practices-for-Your-Data-Architecture-v3-OG-1200x628

Thanks to everyone who joined the Best Practices for Your Data Architecture session on Getting Workloads to Production using CI/CD. You can access the on-demand session recording here, and the code in the Databricks Labs CI/CD Templates Repo. Posted ...

Data Engineering
Reply
  • 2712 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
MadelynM
Databricks Employee
  • 1 kudos

Here's the embedded links list!Jobs scheduling and orchestrationBuilt-in job scheduling: https://docs.databricks.com/jobs.html#schedule-a-job Periodic scheduling of the jobsExecute notebook / jar / Python script / Spark-submitMultitask JobsExecute no...

  • 1 kudos
1 More Replies
RantoB
by • Valued Contributor
  • 2386 Views
  • 2 replies
  • 4 kudos

Resolved! Import a notebook in a Release Pipeline with a Python script

Hi, I would like to import a python file to Databricks with a Azure DevOps Release Pipeline.Within the pipeline I execute a python script which contains this code :import sys import os import base64 import requests   dbw_url = sys.argv[1] # https://a...

Data Engineering
Reply
  • 2386 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

Recently I wrote about alternative way to export/import notebooks in pthon https://community.databricks.com/s/question/0D53f00001TgT52CAF/import-notebook-with-python-script-using-api This way you will get more readable error/message (often it is rela...

  • 4 kudos
1 More Replies
Abhendu
by • New Contributor II
  • 1729 Views
  • 2 replies
  • 0 kudos

Resolved! CICD Databricks

Hi TeamI was wondering if there is a document or step by step process to promote code in CICD across various environments of code repository (GIT/GITHUB/BitBucket/Gitlab) with DBx support? [Without involving code repository merging capability of the ...

Data Engineering
Reply
  • 1729 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Please refer this related thread on CICD in Databricks https://community.databricks.com/s/question/0D53f00001GHVhMCAX/what-are-some-best-practices-for-cicd

  • 0 kudos
1 More Replies
Labels