- 9079 Views
- 9 replies
- 1 kudos
How to use the git CLI in databricks?
After making some changes in my feature branch, I have committed and pushed (to Azure Devops) some work (note I have not yet raised a PR or merge to any other branch). Many of the files I committed are data files and so I would like to reverse the co...
- 9079 Views
- 9 replies
- 1 kudos
- 1 kudos
Any updates on this? We still can't manage to run Git CLI commands from Databricks. Appreciate any input on this!
- 1 kudos
- 11221 Views
- 4 replies
- 0 kudos
Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....
Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...
- 11221 Views
- 4 replies
- 0 kudos
- 0 kudos
@Debayan Mukherjee​ Hello, thank you for your response. please let me know if these are the correct commands to access the file from notebookI can see the files in the repo folderbut I just noticed this. the file I am trying to access the size is 0 b...
- 0 kudos
- 48325 Views
- 7 replies
- 11 kudos
Resolved! Automate the Databricks workflow deployment
Hi everyone,Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?
- 48325 Views
- 7 replies
- 11 kudos
- 11 kudos
Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflowYou can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic w...
- 11 kudos
- 6746 Views
- 7 replies
- 2 kudos
Run Task as Service Principal with Code in Azure DevOps Repo
Hi allI have a task of type Notebook, source is Git (Azure DevOps). This task runs fine with my user, but if I change the Owner to a service principal, I get the following error:Run result unavailable: run failed with error message Failed to checkout...
- 6746 Views
- 7 replies
- 2 kudos
- 2 kudos
@pgruetter​ :To enable a service principal to access a specific Azure DevOps repository, you need to grant it the necessary permissions at both the organization and repository levels.Here are the steps to grant the service principal the necessary per...
- 2 kudos
- 2592 Views
- 2 replies
- 0 kudos
Connect to Azure DevOps repository using service principle
My source code is in the VSTS repository and I am using PAT token to connect VSTS from Azure data bricks notebook and then building packages and installing my cluster. For the production environment, I can't use PAT token, so is there any way to conn...
- 2592 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey everyoneI've been working with Azure DevOps and VSTS repositories, and I can relate to the challenges of connecting them securely. thushar, I understand your concern about using a PAT token for production environments. Fortunately, there is indee...
- 0 kudos
- 9265 Views
- 4 replies
- 3 kudos
Databricks CI/CD Azure DevOps
Hi All. I have a scenario where there are few .sql scripts present in my repo. Is there any way we can execute those SQLs on Databricks via Azure DevOps CI/CD pipeline?Please help.
- 9265 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Divyansh Jain​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 3 kudos
- 1830 Views
- 1 replies
- 2 kudos
Installing private python Azure DevOps repository without revealing personal access token in pyproject.toml
I want to install a .whl file on my Databricks cluster which includes a private Azure DevOps repository as a dependency in its pyproject.toml file, i.e.:[project] name = "test" description = "test_description." version = "0.1.0" authors = [ { name ...
- 1830 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Aaron Kaijser​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 2157 Views
- 3 replies
- 3 kudos
Unable to run jobs with git notebooks
So, in this case our jobs are deployed from our development workspace to our isolated testing workspace via an automated Azure DevOps pipeline. As such, they are created (and thus run as) a service account user.Recently we made the switch to using gi...
- 2157 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Chris Block​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 3 kudos
- 1647 Views
- 2 replies
- 2 kudos
Automated CI code checks using workflows when PR is raised
I'm familiar with Github Actions workflows to automate code checks whenever a PR is raised to a specified branch. For example for Python code, very useful is if unit tests (e.g. pytest), syntax (flake8), and code formatting (black formatter), type h...
- 1647 Views
- 2 replies
- 2 kudos
- 2 kudos
In a typical software development workflow (e.g. Github flow), a feature branch is created based on the master branch for feature development. A notebook can be synced to the feature branch via Github integration. Or a notebook can be exported from D...
- 2 kudos
- 5586 Views
- 11 replies
- 2 kudos
Resolved! DBT Job Type Authenticating to Azure Devops for git_source
we are trying to execute the databricks jobs for dbt task type but it is failing to autheticate to git. Problem is job is created using service principal but service principal don't seem to have access to the repo. few questions we have:1) can we giv...
- 5586 Views
- 11 replies
- 2 kudos
- 2 kudos
Hi @Rahul Samant​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest p...
- 2 kudos
- 5523 Views
- 5 replies
- 0 kudos
IP Access list Databricks
Hello, good morning everyone, I have a problem, I have IP access List activated in my Databricks workspace and I have connections to powerBI and Azure DevOps, in Power BI I have already added all the public IP ranges that Azure gives me, which are ab...
- 5523 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @David Cantos​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 0 kudos
- 2693 Views
- 3 replies
- 1 kudos
Issue with VS Code extension repo.
I successfully installed the extension and connected it to my databricks account. But when I try to select the repo (which already exists under repos in my databricks account) for syncing , I don't see it. My company uses Azure Devops (Git repo) as s...
- 2693 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, was the repo already added?You can check on this : https://learn.microsoft.com/en-us/azure/databricks/repos/git-operations-with-reposAlso, please tag @Debayan Mukherjee​ with your next response which will notify me. Thank you!
- 1 kudos
- 4775 Views
- 5 replies
- 6 kudos
Usage of Azure DevOps System.AccessToken as PAT in Databricks
Hi there! I'm trying to use Azure DevOps Pipeline to automate Azure Databricks Repos API. Im using the following workflow:Get an Access Token for a Databricks Service Principal using a Certificate (which works great)Usage REST Api to generate Git Cre...
- 4775 Views
- 5 replies
- 6 kudos
- 6 kudos
@Adrian Ehrsam​ The PAT limit has been increased to 2048 now. Please check.
- 6 kudos
- 2292 Views
- 3 replies
- 2 kudos
Resolved! Is there a way to tell if a created job is not compliant against configured cluster policies before it runs?
As a DevOps engineer, I want to enforce cluster policies at deployment time when the job is deployed/created, well before it is time to actually use it (i.e. before its scheduled/triggered run time without actually running it).
- 2292 Views
- 3 replies
- 2 kudos
- 2 kudos
Is it not the linked service that defines the kind of cluster created or used for any job?So i believe you could control the configuration via the linked service settings.
- 2 kudos
- 4490 Views
- 3 replies
- 2 kudos
Resolved! Automation creation of Databricks Top Level Repositories
I'm looking to automate the creation of Top Level repositories in Databricks however isn't possible using cli or API if this repo is private repository(Azure DevOps Repository) because require setup the token in user setting.databricks repos create \...
- 4490 Views
- 3 replies
- 2 kudos
- 2 kudos
We have API availabe for repos https://docs.databricks.com/dev-tools/api/latest/repos.html#operation/get-repos also we are currently supporting SP. Step 1: As an admin Create a Service PrincipalUse this API SCIM API 2.0 (ServicePrincipals) | Databric...
- 2 kudos