Databricks Community

Carsten03 · ‎02-19-2024

Hey,

I am using Bitbucket Cloud and I want to connect my repository to Databricks. I am able to connect with my personal app password but what I am looking for is an authentication of a technical user.

I need the integration to point to my dbt repo, which will run as a workflow task and I don't want to use my personal git credentials for that because the service principal is the owner of the pipeline.

Is there a way of using non-personal access tokens to authenticate or is the only way creating a new technical user on Bitbucket?

Thank you!

Kaniz_Fatma · ‎02-19-2024

Hi @Carsten03, Let’s explore the options for authenticating your Databricks integration with Bitbucket Cloud.

Databricks GitHub App (Recommended):
- If you’re using a hosted GitHub account, consider using the Databricks GitHub App for user authentication instead of Personal Access Tokens (PATs).
- Benefits of using the Databricks GitHub App:
  - It uses OAuth 2.0 for user authentication, ensuring strong security.
  - OAuth 2.0 repo traffic is encrypted.
  - Integration is easier and doesn’t require individual tracking of tokens.
  - Token renewal is handled automatically.
  - You can scope the integration to specific attached Git repos, allowing granular control over access.
- To set this up:
  - Link your GitHub account in Databricks by going to User Settings > Linked accounts.
  - Authorize the Databricks GitHub app to act on your behalf for Git operations in Repos (such as cloning a repository).
  - Install and configure the Databricks GitHub app on the GitHub repositories you want to access from Databricks Repos.
  - Note that this feature is not supported in GitHub Enterprise Server; use a personal access token ins...¹.
Bitbucket Cloud Repository Access Token:
- To avoid tying git credentials to an individual user, you can use a Repository Access Token from Bitbucket Cloud.
- However, the Databricks API does not explicitly define how a Repository Access Token should fit, as there is no defined git_username other than the <hash>@bots.
- If you choose this approach, create an app password in Bitbucket Cloud that allows access to your repositories. Record the password.
- In Databricks, go to User Settings > Linked accounts and enter this password to link your Bitbucket ...².
Databricks Personal Access Token (PAT):
- If you prefer to use a PAT, you can set up authentication for the Databricks CLI using a Databricks PAT.
- First, create a PAT for your Databricks workspace user. Then configure the Databricks CLI to use this token ³.

View solution in original post

Kaniz_Fatma · ‎02-19-2024

Hi @Carsten03, Let’s explore the options for authenticating your Databricks integration with Bitbucket Cloud.

Databricks GitHub App (Recommended):
- If you’re using a hosted GitHub account, consider using the Databricks GitHub App for user authentication instead of Personal Access Tokens (PATs).
- Benefits of using the Databricks GitHub App:
  - It uses OAuth 2.0 for user authentication, ensuring strong security.
  - OAuth 2.0 repo traffic is encrypted.
  - Integration is easier and doesn’t require individual tracking of tokens.
  - Token renewal is handled automatically.
  - You can scope the integration to specific attached Git repos, allowing granular control over access.
- To set this up:
  - Link your GitHub account in Databricks by going to User Settings > Linked accounts.
  - Authorize the Databricks GitHub app to act on your behalf for Git operations in Repos (such as cloning a repository).
  - Install and configure the Databricks GitHub app on the GitHub repositories you want to access from Databricks Repos.
  - Note that this feature is not supported in GitHub Enterprise Server; use a personal access token ins...¹.
Bitbucket Cloud Repository Access Token:
- To avoid tying git credentials to an individual user, you can use a Repository Access Token from Bitbucket Cloud.
- However, the Databricks API does not explicitly define how a Repository Access Token should fit, as there is no defined git_username other than the <hash>@bots.
- If you choose this approach, create an app password in Bitbucket Cloud that allows access to your repositories. Record the password.
- In Databricks, go to User Settings > Linked accounts and enter this password to link your Bitbucket ...².
Databricks Personal Access Token (PAT):
- If you prefer to use a PAT, you can set up authentication for the Databricks CLI using a Databricks PAT.
- First, create a PAT for your Databricks workspace user. Then configure the Databricks CLI to use this token ³.

Carsten03 · ‎02-20-2024

Hi @Kaniz_Fatma,

thank you for your response! With this link you provided, I was able to authenticate with Bitbucket Cloud. The solution was to use x-token-auth as a username. I have tried with the generated email address before which didn't work. Thank you! Now I am facing the next issue.

I wanted to use the repository access to run dbt via workflows. I have added the git credential via terraform and checked with the cli that it is available, but I now I get the following error message:

[RepositoryCheckoutFailed] Failed to checkout Git repository: PERMISSION_DENIED: Missing Git provider credentials. Go to User Settings > Git Integration to set up your Git credentials

I want to run the workflow with a service principal as an owner and therefore don't want to set the git credentials for my personal user. How can the service principal get access to the git credential / repository?