cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Jiri_Koutny
by • New Contributor III
  • 5675 Views
  • 11 replies
  • 3 kudos

Delay in files update on filesystem

Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If the...

Data Engineering
Reply
  • 5675 Views
  • 11 replies
  • 3 kudos
View Replies
Latest Reply
Irka
New Contributor II
  • 3 kudos

Is there a solution to this?BTW, the "ls" command trick didn't work for me

  • 3 kudos
10 More Replies
Arby
by • New Contributor II
  • 10711 Views
  • 4 replies
  • 0 kudos

Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....

Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...

icon
Data Engineering
Reply
  • 10711 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Arby
New Contributor II
  • 0 kudos

@Debayan Mukherjee​ Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...

  • 0 kudos
3 More Replies
dannythermadom
by • New Contributor III
  • 4393 Views
  • 6 replies
  • 7 kudos

Dbutils.notebook.run command not working with /Repos/

I have two github repo configured in Databricks Repos folder. repo_1 is run using a job and repo_2 is run/called from repo_1 using Dbutils.notebook.run command. dbutils.notebook.run("/Repos/repo_2/notebooks/notebook", 0, args)i am getting the follo...

Data Engineering
Reply
  • 4393 Views
  • 6 replies
  • 7 kudos
View Replies
Latest Reply
cubanDataDude
New Contributor II
  • 7 kudos

I am having a similar issue...  ecw_staging_nb_List = ['/Workspace/Repos/PRIMARY/UVVC_DATABRICKS_EDW/silver/nb_UPSERT_stg_ecw_insurance',                 '/Repos/PRIMARY/UVVC_DATABRICKS_EDW/silver/nb_UPSERT_stg_ecw_facilitygroups'] Adding workspace d...

  • 7 kudos
5 More Replies
michael_mehrten
by • New Contributor III
  • 27443 Views
  • 25 replies
  • 14 kudos

Resolved! How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Databricks Repos best-practices recommend using the Repos REST API to update a repo via your git provider. The REST API requires authentication, which can be done one of two ways:A user / personal access tokenA service principal access tokenUsing a u...

Data Engineering
Reply
  • 27443 Views
  • 25 replies
  • 14 kudos
View Replies
Latest Reply
martindlarsson
New Contributor III
  • 14 kudos

Having the exact same problem. Did you find a solution @michael_mehrten ?In my case Im using a managed identity so the solution some topics suggest on generating an access token from a Entra ID service principal is not applicable.

  • 14 kudos
24 More Replies
aseufert
by • New Contributor III
  • 9722 Views
  • 2 replies
  • 3 kudos

Git Stash

Looked through some previous posts and documentation and couldn't find anything related to use of Git stash in Databricks Repos. Perhaps I missed it. I also don't see an option in the UI.Does anyone know if there's a way to stash changes either in th...

Data Engineering
Reply
  • 9722 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
javierbg
New Contributor III
  • 3 kudos

This is actually a big hurdle when trying to switch between working in two different branches, it would be a welcome addition to the Databricks IDE.

  • 3 kudos
1 More Replies
Serhii
by • Contributor
  • 7476 Views
  • 7 replies
  • 4 kudos

Resolved! Saving complete notebooks to GitHub from Databricks repos.

When saving notebook to GiHub repo, it is stripped to Python source code. Is it possible to save it in the ipynb formt?

Data Engineering
Reply
  • 7476 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
GlennStrycker
New Contributor III
  • 4 kudos

When I save+commit+push my .ipynb file to my linked git repo, I noticed that only the cell inputs are saved, not the output.  This differs from the .ipynb file I get when I choose "File / Export / iPython Notebook".  Is there a way to save the cell o...

  • 4 kudos
6 More Replies
alvaro_databric
by • New Contributor III
  • 13757 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks Python Import Bug

Hello,I am reaching to the community in order to shed some light into a Bug I have been encountering recently. The two setups are as follows:SETUP-1 (WORKS):Python Notebook in Workspace FileSystem (this is Shared/folder/...)Python custom wheel librar...

Data Engineering
Reply
  • 13757 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
alvaro_databric
New Contributor III
  • 1 kudos

Solution has comed with an update, as stated in Work with Python and R modules | Databricks on AWS, In Databricks Runtime 13.0 and above, directories added to the Python sys.path are automatically distributed to all executors in the cluster. In Datab...

  • 1 kudos
2 More Replies
Priyag1
by • Honored Contributor II
  • 2178 Views
  • 2 replies
  • 4 kudos

docs.databricks.com

Databricks new Release : Full-page workspace browser includes ReposDatabricks plans to enable the full-page workspace browser experience that unifies Workspace and Repos by default. You’ll browse content in Databricks Repos alongside your workspace c...

Data Engineering
Reply
  • 2178 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
bharats
New Contributor III
  • 4 kudos

Thanks for the update

  • 4 kudos
1 More Replies
reachbharathan
by • New Contributor III
  • 2896 Views
  • 3 replies
  • 4 kudos

Resolved! How to checkout specific commit version via databricks UI

I have integrated gitlab with my azure databricks repo, I am able to push and pull commits from the databricks UI, I want to checkout to a specific commit version via databricks UI.Note: I am aware that via the gitlab i have checkout to specific vers...

Data Engineering
Reply
  • 2896 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
reachbharathan
New Contributor III
  • 4 kudos

After getting more context on databricks repo in details,Currently databricks doesn't support checkout of repo to specific commit.databricks provides only limited functionality mentioned belowAdd a repo and connect remotely laterClone a repo connecte...

  • 4 kudos
2 More Replies
Tsar
by • New Contributor III
  • 11180 Views
  • 10 replies
  • 12 kudos

Limitations with UDFs wrapping modules imported via Repos files?

We have been importing custom module wheel files from our AzDevOps repository. We are pushing to use the Databricks Repos arbitrary files to simplify this but it is breaking our spark UDF that wraps one of the functions in the library with a ModuleNo...

Data Engineering
Reply
  • 11180 Views
  • 10 replies
  • 12 kudos
View Replies
Latest Reply
Scott_B
New Contributor III
  • 12 kudos

If your notebook is in the same Repo as the module, this should work without any modifications to the sys path.If your notebook is not in the same Repo as the module, you may need to ensure that the sys path is correct on all nodes in your cluster th...

  • 12 kudos
9 More Replies
Simmy
by • New Contributor II
  • 2223 Views
  • 3 replies
  • 1 kudos

Databricks Repos are pushing all changes to GitHub

When I make any changes within a repo, when I go to commit and push to GitHub if I uncheck any changes that I don't want pushed, still get pushed to Github. Any help would be appreciated

Data Engineering
Reply
  • 2223 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Simmy
New Contributor II
  • 1 kudos

Hi @Vidula Khanna​ Problem is now resolved thanks, didn't have to do anything different, the functionality just started working as expected.

  • 1 kudos
2 More Replies
pabloaus
by • New Contributor III
  • 5289 Views
  • 2 replies
  • 4 kudos

Resolved! How to read sql file from a Repo to string

I am trying to read a sql file in the repo to string. I have triedwith open("/Workspace/Repos/xx@***.com//file.sql","r") as queryFile: queryText = queryFile.read()And I get following error.[Errno 1] Operation not permitted: '/Workspace/Repos/***@*...

Data Engineering
Reply
  • 5289 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
Senthil1
Contributor
  • 4 kudos

I checked in my unity_catalog enabled cluster, i am able to access the @repos file to read and display

  • 4 kudos
1 More Replies
martikev
by • New Contributor II
  • 1461 Views
  • 2 replies
  • 1 kudos

How to avoid losing dbutils.widgets parameters when connecting databricks repos folder with azure dev ops?

Hi guys We want to move from databricks workspace to databricks repos. Now when we create a new folder under repos and connect to our azure dev ops repository all our dbutils.widget parameters are lost. How to avoid this? We want to fully depend on a...

Data Engineering
Reply
  • 1461 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Kevin Peter M. Marti​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 1 kudos
1 More Replies
tompile
by • New Contributor III
  • 4208 Views
  • 6 replies
  • 9 kudos

Resolved! Is it possible to make use of pygit2 or GitPython packages to reference git repositories from within databricks?

I am making use of repos in databricks and am trying to reference the current git branch from within the notebook session.For example:from pygit2 import Repositoryrepo = Repository('/Workspace/Repos/user@domain/repository')The code above throws an er...

Data Engineering
Reply
  • 4208 Views
  • 6 replies
  • 9 kudos
View Replies
Latest Reply
niburg123
New Contributor III
  • 9 kudos

You cannot use this as far as i know, but you can put a workaround in a notebook if you are calling code from your repo via a notebook:repo_path = "/Repos/xyz_repo_path/xyz_repo_name"repo_path_fs = "/Workspace" + repo_pathrepo_branch = "main"def chec...

  • 9 kudos
5 More Replies
shubhamb
by • New Contributor III
  • 3978 Views
  • 1 replies
  • 3 kudos

How to fetch environmental variables saved in one notebook into another notebook in Databricks Repos and Notebooks

I have this config.py file which is used to store environmental variablesPUSH_API_ACCOUNT_ID = '*******' PUSH_API_PASSCODE = '***********************'I am using this to fetch the variables and use it in my file.py import sys   sys.path.append("..") ...

Data Engineering
Reply
  • 3978 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Shubham Biswas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...

  • 3 kudos
Labels