cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
dannythermadom
by • New Contributor III
  • 4340 Views
  • 6 replies
  • 7 kudos

Dbutils.notebook.run command not working with /Repos/

I have two github repo configured in Databricks Repos folder. repo_1 is run using a job and repo_2 is run/called from repo_1 using Dbutils.notebook.run command. dbutils.notebook.run("/Repos/repo_2/notebooks/notebook", 0, args)i am getting the follo...

Data Engineering
Reply
  • 4340 Views
  • 6 replies
  • 7 kudos
View Replies
Latest Reply
cubanDataDude
New Contributor II
  • 7 kudos

I am having a similar issue...  ecw_staging_nb_List = ['/Workspace/Repos/PRIMARY/UVVC_DATABRICKS_EDW/silver/nb_UPSERT_stg_ecw_insurance',                 '/Repos/PRIMARY/UVVC_DATABRICKS_EDW/silver/nb_UPSERT_stg_ecw_facilitygroups'] Adding workspace d...

  • 7 kudos
5 More Replies
michael_mehrten
by • New Contributor III
  • 27096 Views
  • 25 replies
  • 14 kudos

Resolved! How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Databricks Repos best-practices recommend using the Repos REST API to update a repo via your git provider. The REST API requires authentication, which can be done one of two ways:A user / personal access tokenA service principal access tokenUsing a u...

Data Engineering
Reply
  • 27096 Views
  • 25 replies
  • 14 kudos
View Replies
Latest Reply
martindlarsson
New Contributor III
  • 14 kudos

Having the exact same problem. Did you find a solution @michael_mehrten ?In my case Im using a managed identity so the solution some topics suggest on generating an access token from a Entra ID service principal is not applicable.

  • 14 kudos
24 More Replies
pgagliardi
by • New Contributor II
  • 1828 Views
  • 1 replies
  • 2 kudos

Latest pushed code is not taken into account by Notebook

Hello, I cloned a repo my_repo in the Dataricks space Repos.Inside my_repo, I created a notebook new_experiment where I can import functions from my_repo, which is really handy. When I want to modify a function in my_repo, I open my local IDE, do the...

Data Engineering
Reply
  • 1828 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Jnguyen
Databricks Employee
  • 2 kudos

Use  %reload_ext autoreload instead, it will do your expected behavior.You just need to run it once, like %load_ext autoreload %autoreload 2

  • 2 kudos
jfarmer
by • New Contributor II
  • 5358 Views
  • 3 replies
  • 1 kudos

PermissionError / Operation not Permitted with Files-in-Repos

I've been running a notebook using files-in-repo. Previously this has worked fine. I'm unsure what's changed (I was testing integration with DCS on older runtimes, but don't think I made any persistent changes)--but now it's throwing an error (always...

image image
Data Engineering
Reply
  • 5358 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
_carleto_
New Contributor II
  • 1 kudos

Hi @jfarmer , did you solved this issue? I'm having exactly the same challenge.Thanks!

  • 1 kudos
2 More Replies
yopbibo
by • Contributor II
  • 10807 Views
  • 6 replies
  • 0 kudos

Resolved! How to copy the content of a repos, in the workspace/shared, automatically, daily?

How to copy the content of a repos, in the workspace/shared, automatically, daily?Purpose here is to bring some notebooks, in shared, available to all workspace users, without requesting users to use REPOS.

Data Engineering
Reply
  • 10807 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
citizenkrank
New Contributor II
  • 0 kudos

Alternatively, you can schedule a notebook with the following cell:%sh cp -r /Workspace/Repos/username/repo_name /Workspace/SharedPlease note that you'll have to update (i.e. pull) the repo manually if you've updated it somewhere else (although you c...

  • 0 kudos
5 More Replies
cmilligan
by • Contributor II
  • 808 Views
  • 1 replies
  • 1 kudos

Return notebook path from job that is run remotely from the repo

I'm wanting to set up some email alerts for issues in the data as a part of a job run. I am wanting to point the user to the notebook that the issue occurred in. I think this would be simple enough but another layer is that the job is going to be run...

Data Engineering
Reply
  • 808 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, Could you please clarify what do you mean by return the file from the remote repo?Please tag @Debayan​ with your next response which will notify me, Thank you!

  • 1 kudos
William_Scardua
by • Valued Contributor
  • 1342 Views
  • 1 replies
  • 0 kudos

REPOS change my notebook format

Hi guys,I have some notebooks with REPOS but I noticed that REPOS changed my notebook format to .py because of this my Azure Data Factory no longer recognizes the notebook (.py)Have any ideia to convert that .py to databricks format ?

Screenshot 2023-05-30 at 20.39.02
Data Engineering
Reply
  • 1342 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

that is odd. repos is merely another location (linked to git).You can copy/paste the code inside the py file into a notebook, or convert them using online tools or python libraries (like py2ipynb).

  • 0 kudos
pantelis_mare
by • Contributor III
  • 27164 Views
  • 30 replies
  • 15 kudos

Resolved! Repos configuration for Azure Service Principal

Hello community!I would like to update a repo from within my Azure DevOps release pipeline. In the pipeline I generate a token using a AAD Service Principal as recommended, and I setup the databricks api using that token.When I pass the databricks re...

Data Engineering
Reply
  • 27164 Views
  • 30 replies
  • 15 kudos
View Replies
Latest Reply
xiangzhu
Contributor III
  • 15 kudos

traditional PAT may have long lifespn, but the new SP feature uses an AAD token which should have a much shorter lifespqn, maybe around one hour, this could be a limiting factor.However, I haven't tested this yet, so these are merely hypotheses.​Neve...

  • 15 kudos
29 More Replies
thib
by • New Contributor III
  • 4908 Views
  • 3 replies
  • 2 kudos

Can we use multiple git repos for a job running multiple tasks?

I have a job running multiple tasks :Task 1 runs a machine learning pipeline from git repo 1Task 2 runs an ETL pipeline from git repo 1Task 2 is actually a generic pipeline and should not be checked in repo 1, and will be made available in another re...

image
Data Engineering
Reply
  • 4908 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
trijit
New Contributor II
  • 2 kudos

The way to go about this would be to create Databricks repos in the workspace and then use that in the task formation. This way we can refer multiple repos in different tasks.

  • 2 kudos
2 More Replies
gentresh
by • New Contributor III
  • 2816 Views
  • 1 replies
  • 0 kudos

Running terraform plan; databricks throws a "Permission_Denied: Missing required permissions [View] on node with ID "1759335429158542"

To give you a little bit of background:We use terraform to deploy a resource group with multiple Azure services Terraform leverages an Azure Service Principal that has Owner rights to the Azure subscriptionThis way, databricks is also deployed. We al...

Data Engineering
Reply
  • 2816 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Gent Reshtani​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...

  • 0 kudos
youssefmrini
by • Databricks Employee
  • 1095 Views
  • 1 replies
  • 0 kudos

Resolved! Does repos support Ipynb ?

Data Engineering
Reply
  • 1095 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can ...

  • 0 kudos
Simmy
by • New Contributor II
  • 2196 Views
  • 3 replies
  • 1 kudos

Databricks Repos are pushing all changes to GitHub

When I make any changes within a repo, when I go to commit and push to GitHub if I uncheck any changes that I don't want pushed, still get pushed to Github. Any help would be appreciated

Data Engineering
Reply
  • 2196 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Simmy
New Contributor II
  • 1 kudos

Hi @Vidula Khanna​ Problem is now resolved thanks, didn't have to do anything different, the functionality just started working as expected.

  • 1 kudos
2 More Replies
m2chrisp
by • New Contributor II
  • 6838 Views
  • 3 replies
  • 0 kudos

Resolved! Deploying Databricks via ARM, then configuring via databricks-cli: creating a repo results in INVALID_STATE error.

Hi,I'm using databricks-cli to configure a newly-deployed Azure Databricks instance. The ARM deployment works fine, and then the databricks cli commands to create a secrets scope and add users also works just fine.Then I add a GitCredential to Datab...

Data Engineering
Reply
  • 6838 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Chris Platts​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Thanks and Regards

  • 0 kudos
2 More Replies
MSD
by • New Contributor
  • 4260 Views
  • 5 replies
  • 4 kudos

Resolved! Databricks Community version - Unable to clone a public git repository

Databricks Community version - Unable to clone a public git repository, as the 'Repository' tab that should appear below 'Workspace' tab on the portal does not appear and I am not aware of any alternate method. I have referred to some documents on th...

community_repository
Data Engineering
Reply
  • 4260 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Jay Kumar​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wi...

  • 4 kudos
4 More Replies
Labels