cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Arby
by New Contributor II
  • 6991 Views
  • 4 replies
  • 0 kudos

Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....

Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...

icon
  • 6991 Views
  • 4 replies
  • 0 kudos
Latest Reply
Arby
New Contributor II
  • 0 kudos

@Debayan Mukherjee​ Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...

  • 0 kudos
3 More Replies
joao_vnb
by New Contributor III
  • 30852 Views
  • 7 replies
  • 11 kudos

Resolved! Automate the Databricks workflow deployment

Hi everyone,Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?

  • 30852 Views
  • 7 replies
  • 11 kudos
Latest Reply
asingamaneni
New Contributor II
  • 11 kudos

Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflowYou can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic w...

  • 11 kudos
6 More Replies
pgruetter
by Contributor
  • 4514 Views
  • 7 replies
  • 2 kudos

Run Task as Service Principal with Code in Azure DevOps Repo

Hi allI have a task of type Notebook, source is Git (Azure DevOps). This task runs fine with my user, but if I change the Owner to a service principal, I get the following error:Run result unavailable: run failed with error message Failed to checkout...

  • 4514 Views
  • 7 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@pgruetter​ :To enable a service principal to access a specific Azure DevOps repository, you need to grant it the necessary permissions at both the organization and repository levels.Here are the steps to grant the service principal the necessary per...

  • 2 kudos
6 More Replies
Oliver_Angelil
by Valued Contributor II
  • 6155 Views
  • 8 replies
  • 1 kudos

How to use the git CLI in databricks?

After making some changes in my feature branch, I have committed and pushed (to Azure Devops) some work (note I have not yet raised a PR or merge to any other branch). Many of the files I committed are data files and so I would like to reverse the co...

  • 6155 Views
  • 8 replies
  • 1 kudos
Latest Reply
Kayla
Contributor III
  • 1 kudos

I'm also curious about this question - does anyone have an answer? Being able to use the full repertoire of git commands inside Databricks would be quite useful.

  • 1 kudos
7 More Replies
thushar
by Contributor
  • 2035 Views
  • 3 replies
  • 0 kudos

Connect to Azure DevOps repository using service principle

My source code is in the VSTS repository and I am using PAT token to connect VSTS from Azure data bricks notebook and then building packages and installing my cluster. For the production environment, I can't use PAT token, so is there any way to conn...

  • 2035 Views
  • 3 replies
  • 0 kudos
Latest Reply
martinez
New Contributor III
  • 0 kudos

Hey everyoneI've been working with Azure DevOps and VSTS repositories, and I can relate to the challenges of connecting them securely. thushar, I understand your concern about using a PAT token for production environments. Fortunately, there is indee...

  • 0 kudos
2 More Replies
DJey
by New Contributor III
  • 4745 Views
  • 4 replies
  • 3 kudos

Databricks CI/CD Azure DevOps

Hi All. I have a scenario where there are few .sql scripts present in my repo. Is there any way we can execute those SQLs on Databricks via Azure DevOps CI/CD pipeline?Please help.

  • 4745 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Divyansh Jain​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...

  • 3 kudos
3 More Replies
Kaijser
by New Contributor II
  • 1296 Views
  • 1 replies
  • 2 kudos

Installing private python Azure DevOps repository without revealing personal access token in pyproject.toml

I want to install a .whl file on my Databricks cluster which includes a private Azure DevOps repository as a dependency in its pyproject.toml file, i.e.:[project] name = "test" description = "test_description." version = "0.1.0" authors = [ { name ...

  • 1296 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Aaron Kaijser​  Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
cblock
by New Contributor III
  • 1577 Views
  • 3 replies
  • 3 kudos

Unable to run jobs with git notebooks

So, in this case our jobs are deployed from our development workspace to our isolated testing workspace via an automated Azure DevOps pipeline. As such, they are created (and thus run as) a service account user.Recently we made the switch to using gi...

  • 1577 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Chris Block​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 3 kudos
2 More Replies
Oliver_Angelil
by Valued Contributor II
  • 1155 Views
  • 2 replies
  • 2 kudos

Automated CI code checks using workflows when PR is raised

I'm familiar with Github Actions workflows to automate code checks whenever a PR is raised to a specified branch. For example for Python code, very useful is if unit tests (e.g. pytest), syntax (flake8), and code formatting (black formatter), type h...

  • 1155 Views
  • 2 replies
  • 2 kudos
Latest Reply
Priyag1
Honored Contributor II
  • 2 kudos

In a typical software development workflow (e.g. Github flow), a feature branch is created based on the master branch for feature development. A notebook can be synced to the feature branch via Github integration. Or a notebook can be exported from D...

  • 2 kudos
1 More Replies
rsamant07
by New Contributor III
  • 3910 Views
  • 11 replies
  • 2 kudos

Resolved! DBT Job Type Authenticating to Azure Devops for git_source

we are trying to execute the databricks jobs for dbt task type but it is failing to autheticate to git. Problem is job is created using service principal but service principal don't seem to have access to the repo. few questions we have:1) can we giv...

  • 3910 Views
  • 11 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rahul Samant​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest p...

  • 2 kudos
10 More Replies
dcantos
by New Contributor II
  • 3756 Views
  • 5 replies
  • 0 kudos

IP Access list Databricks

Hello, good morning everyone, I have a problem, I have IP access List activated in my Databricks workspace and I have connections to powerBI and Azure DevOps, in Power BI I have already added all the public IP ranges that Azure gives me, which are ab...

  • 3756 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @David Cantos​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
4 More Replies
Sorush
by New Contributor II
  • 2127 Views
  • 3 replies
  • 1 kudos

Issue with VS Code extension repo.

I successfully installed the extension and connected it to my databricks account. But when I try to select the repo (which already exists under repos in my databricks account) for syncing , I don't see it. My company uses Azure Devops (Git repo) as s...

  • 2127 Views
  • 3 replies
  • 1 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, was the repo already added?You can check on this : https://learn.microsoft.com/en-us/azure/databricks/repos/git-operations-with-reposAlso, please tag @Debayan Mukherjee​ with your next response which will notify me. Thank you!

  • 1 kudos
2 More Replies
arsamkull
by New Contributor III
  • 3558 Views
  • 5 replies
  • 6 kudos

Usage of Azure DevOps System.AccessToken as PAT in Databricks

Hi there! I'm trying to use Azure DevOps Pipeline to automate Azure Databricks Repos API. Im using the following workflow:Get an Access Token for a Databricks Service Principal using a Certificate (which works great)Usage REST Api to generate Git Cre...

  • 3558 Views
  • 5 replies
  • 6 kudos
Latest Reply
Srihasa_Akepati
New Contributor III
  • 6 kudos

@Adrian Ehrsam​ The PAT limit has been increased to 2048 now. Please check.

  • 6 kudos
4 More Replies
successhawk
by New Contributor II
  • 1642 Views
  • 4 replies
  • 3 kudos

Resolved! Is there a way to tell if a created job is not compliant against configured cluster policies before it runs?

As a DevOps engineer, I want to enforce cluster policies at deployment time when the job is deployed/created, well before it is time to actually use it (i.e. before its scheduled/triggered run time without actually running it).

  • 1642 Views
  • 4 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Nathan Hawk​, We haven’t heard from you since the last response from @nafri A​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Als...

  • 3 kudos
3 More Replies
alefred
by New Contributor II
  • 3438 Views
  • 5 replies
  • 2 kudos

Resolved! Automation creation of Databricks Top Level Repositories

I'm looking to automate the creation of Top Level repositories in Databricks however isn't possible using cli or API if this repo is private repository(Azure DevOps Repository) because require setup the token in user setting.databricks repos create \...

  • 3438 Views
  • 5 replies
  • 2 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 2 kudos

We have API availabe for repos https://docs.databricks.com/dev-tools/api/latest/repos.html#operation/get-repos also we are currently supporting SP. Step 1: As an admin Create a Service PrincipalUse this API SCIM API 2.0 (ServicePrincipals) | Databric...

  • 2 kudos
4 More Replies
Labels