cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Ullsokk
by • New Contributor III
  • 2734 Views
  • 4 replies
  • 0 kudos

Running notebook from another notebook does not work when running notebook from github actions

I have a setup-notebook that users %run to run a series of notebooks. The notebook is in the root folder of my repo. In a subfolder I have several notebooks I want to run. If I run the notebook in databricks, the relative paths work (%run "./subfolde...

Data Engineering
Reply
  • 2734 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 0 kudos

It seems that the directory is different under GitHub actions. Not sure what exactly mentioned GitHub actions do and which one it is. Maybe you can share GitHub action used.

  • 0 kudos
3 More Replies
Ulf
by • New Contributor II
  • 1211 Views
  • 1 replies
  • 0 kudos

Github and task integration

I have the same problem as described in this post (https://community.databricks.com/s/question/0D58Y00009ObQgdSAF/running-jobs-using-notebooks-in-a-remote-azure-devops-services-repos-git-repository-is-generating-notebook-not-found-error) and get this...

Data Engineering
Reply
  • 1211 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi,Could you please check and let us know if this helps. https://community.databricks.com/s/question/0D53f00001GHVTNCA5/notebook-path-cant-be-in-dbfs

  • 0 kudos
sfalquier
by • New Contributor II
  • 2379 Views
  • 3 replies
  • 0 kudos

HTTP 403 on git-credentials API

Hi,I am trying to set git credentials for my service principal. I follow the process described here but I get a 403 error when making the POST request to ${DATABRICKS_HOST}/api/2.0/git-credentials with service principal token.By the way, I also canno...

Data Engineering
Reply
  • 2379 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Vivian_Wilfred
Databricks Employee
  • 0 kudos

Hi @Sébastien FALQUIER​ it works for me, there are no restrictions. Maybe the PAT token you generated for the service principle got expired. Can you generate a new token and try to run GET/git-credentials API?How are you creating PAT for service prin...

  • 0 kudos
2 More Replies
JoeS
by • New Contributor III
  • 6139 Views
  • 1 replies
  • 1 kudos

When will Github Copilot be available in the Databricks IDE?

It's been quite difficult to stay in VSCode while developing data science experiments and tooling for Databricks. Our team would like to have Github Copilot for the databricks IDE.

Data Engineering
Reply
  • 6139 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Joe Shull​ Does @Kaniz Fatma​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
KrishZ
by • Contributor
  • 1177 Views
  • 2 replies
  • 0 kudos

Where to report a bug with Databricks ?

I have in issue in Pyspark.Pandas to report. Is there a github or some forum where I can register my issue?Here's the issue

Data Engineering
Reply
  • 1177 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Does @Debayan Mukherjee​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
KrishZ
by • Contributor
  • 2617 Views
  • 2 replies
  • 1 kudos

Where to report a bug with Databricks ?

I have in issue in Pyspark.Pandas to report.Is there a github or some forum where I can register my issue?Here's the issue

Data Engineering
Reply
  • 2617 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, @Krishna Zanwar​ Could you please raise a support case to report the bug. Please refer https://docs.databricks.com/resources/support.html to engage with Databricks Support.

  • 1 kudos
1 More Replies
Philip_Budbee
by • New Contributor II
  • 1395 Views
  • 0 replies
  • 1 kudos

Github workflow integration error

Hi,We have a working Github integration in place for our production workspace which is running 14 different jobs that are scheduled during different intervals, but throughout the entire day.The issue over the past 3-4 weeks that we have encountered i...

Data Engineering
Reply
  • 1395 Views
  • 0 replies
  • 1 kudos
Kit
by • New Contributor III
  • 4792 Views
  • 7 replies
  • 1 kudos

Resolved! Can't run a job that use GitHub as source

I have a list of jobs that are using the code in GitHub as source.Everything worked fine until yesterday. Yesterday, I noticed that all the job that were using GitHub as source were failing. Because of the following error: ``` Run result unavailable:...

Data Engineering
Reply
  • 4792 Views
  • 7 replies
  • 1 kudos
View Replies
Latest Reply
User16766737456
Databricks Employee
  • 1 kudos

Just an update, to round this out. We investigated further internally, and found that although we have a cleanup process in place to remove the internal repos that are being checked out for workflows, it was failing to catch up due to the sheer volum...

  • 1 kudos
6 More Replies
GC-James
by • Contributor II
  • 2364 Views
  • 2 replies
  • 3 kudos

Resolved! Working locally then moving to databricks

Hello DataBricks,Struggling with a workflow issue and wondering if anyone can help. I am developing my project in R and sometimes Python locally on my laptop, and committing the files to a git repo. I can then clone that repo in databricks, and *see*...

Data Engineering
Reply
  • 2364 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

This is separate script which than need to be run from notebook (or job). I am not using R but in Python and Scala it works the same. In Python I am just importing it in notebook ("from folder_structure import myClass") in R probably similar. There ...

  • 3 kudos
1 More Replies
MadelynM
by • Databricks Employee
  • 1072 Views
  • 0 replies
  • 1 kudos

vimeo.com

Repos let you use Git functionality such as cloning a remote repo, managing branches, pushing and pulling changes and visually comparing differences upon commit. Here's a quick video (3:56) on setting up a repo for Databricks on AWS. Pre-reqs: Git in...

Data Engineering
Reply
  • 1072 Views
  • 0 replies
  • 1 kudos
EvandroLippert_
by • New Contributor
  • 1787 Views
  • 1 replies
  • 0 kudos

Conflict with bitbucket and github credentials

I'm migrating my files from Bitbucket to Github, but every time that I need to clone something from bitbucket and send it to GitHub, I need to create a new token to integrate the tools. It seems that when you save a Github credential, it overrides t...

Data Engineering
Reply
  • 1787 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
alexott
Databricks Employee
  • 0 kudos

Cross-posting my answer from StackOverflow:Unfortunately right now it works only with a single Git provider. It looks like that you're linking individual notebooks into Git repository. You can simplify things by cloning the Bitbucket repository(-ies)...

  • 0 kudos
User16137833804
by • Databricks Employee
  • 1550 Views
  • 1 replies
  • 1 kudos

Once I set up the Git Server Proxy, what would be the best way to set alerts in case the Cluster Proxy goes down?

Data Engineering
Reply
  • 1550 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

You could have the single node cluster where proxy is installed monitored by one of the tools like cloudwatch, azure monitor, datadog etc and have it configured to send alerts on node failure

  • 1 kudos
User16753724663
by • Valued Contributor
  • 1856 Views
  • 1 replies
  • 1 kudos

Download private repo from GitHub Enterprise in Databricks notebook

We are trying to download our repository which is hosted on GitHub Enterprise to use its python libraries in our notebooks.Earlier we had issues with downloading our repository using the repos feature in Databricks platform since only notebooks can b...

Data Engineering
Reply
  • 1856 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
User16753724663
Valued Contributor
  • 1 kudos

To fix the issue, we need to pass the token in the header itself git clone https://<token>:x-oauth-basic@github.com/owner/repo.gitExample:%sh   git clone https://<token>@github.com/darshanbargal4747/databricks.git

  • 1 kudos
User16790091296
by • Contributor II
  • 2530 Views
  • 1 replies
  • 0 kudos

Resolved! How can I use a Python function defined in my git-repo module within the DB notebook?

I have a function within a module in my git-repo. I want to import that to my DB notebook - how can I do that?

Data Engineering
Reply
  • 2530 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aladda
Databricks Employee
  • 0 kudos

Databricks Repos allows you to sync your work in Databricks with a remote Git repository. This makes it easier to implement development best practices. Databricks supports integrations with GitHub, Bitbucket, and GitLab. Using Repos you can bring you...

  • 0 kudos
Labels