- 6910 Views
- 5 replies
- 0 kudos
Move whole workflow from Dev to Prod
I have a workflow created in Dev, now I want to move the whole thing to prod and schedule it. The workflow has multiple notebooks, dependent libraries, parameters and such. How to move the whole thing to prod, instead of moving each notebooks and rec...
- 6910 Views
- 5 replies
- 0 kudos
- 0 kudos
I ended up creating a python script to just do the export, here is the code below.It will match on Job name, if it matches it will update otherwise it will import. import requests source_token = '' source_instance = 'adb-000000000000000.00.azuredata...
- 0 kudos
- 42923 Views
- 7 replies
- 11 kudos
Resolved! Automate the Databricks workflow deployment
Hi everyone,Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?
- 42923 Views
- 7 replies
- 11 kudos
- 11 kudos
Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflowYou can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic w...
- 11 kudos
- 11937 Views
- 12 replies
- 9 kudos
PyPI library sometimes doesn't install during workflow execution
I have a workflow that is running upon a job cluster and contains a task that requires prophet library from PyPI:{ "task_key": "my_task", "depends_on": [ { "task_key": "<...>...
- 11937 Views
- 12 replies
- 9 kudos
- 9 kudos
Hey @Eugene Bikkinin Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 9 kudos
- 14593 Views
- 5 replies
- 6 kudos
Workflow service principle owned can't checkout git repository
I am trying to deploy a workflow where the owner is a service principal and I am using git integration (backend with azure devops), when I run the workflow it says that it doesn't have permissions to checkout the repo.run failed with error message F...
- 14593 Views
- 5 replies
- 6 kudos
- 6 kudos
Hi @Giuseppe Grieco Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us ...
- 6 kudos
- 3328 Views
- 2 replies
- 0 kudos
Resolved! Error updating workflow, webhook not found?
I have no idea what this error means or what it could mean. When I'm trying to save a workflow I get a popup saying this:
- 3328 Views
- 2 replies
- 0 kudos
- 0 kudos
I had the same issue, thanks for the info! Apparently it's also possible to fix it by removing all the actual notification in the interface (the bugged one is not displayed, but if you remove everything for some reason it removes the bugged one too)....
- 0 kudos
- 2586 Views
- 4 replies
- 0 kudos
How to build master workflow for all the jobs present in workflow using databricks?
Suppose there are multiple job have been created using databricks workflow, now the requirement is to make one master workflow to trigger all the workflow depending on different condition like: some are supposed to trigger on daily basis, some on mon...
- 2586 Views
- 4 replies
- 0 kudos
- 0 kudos
@Ria Hi , This feature was in development when I attended last Quarter Roadmap and I thought it is available in the latest versions or could be even in Private Preview. You can check with your Databricks Solution Architect. Even if not now, could be ...
- 0 kudos
- 5053 Views
- 9 replies
- 14 kudos
Resolved! Pipeline workflow dude
Hi! I have a problem. I'm using an autoloader to ingest data from raw to a Delta Lake, but when my pipeline starts, I want to apply the pipeline only to the new data. The autoloader ingests data into the Delta Lake, but now, how can I distinguish the...
- 5053 Views
- 9 replies
- 14 kudos
- 14 kudos
Hi @Alejandro Piury Pinzón We haven't heard from you since the last response from @Tyler Retzlaff , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be he...
- 14 kudos
- 952 Views
- 1 replies
- 1 kudos
Mysterious simultaneous long-running Databricks Workflows
Hi,This happened across 4x seemingly unrelated workflows at the same time of the day - all 4x workflows eventually completed successfully. It appeared that all workflows sat idling despite triggering via the Jobs API. The two symptoms I have observed...
- 952 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Timothy Lin Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 15453 Views
- 6 replies
- 15 kudos
Job cluster vs All purpose cluster
Environment: AzureI've a workflow that takes approximately a minute to execute and I want to run the job every 2 minutes.. All purpose cluster:On attaching all purpose cluster to the job, it takes approx. 60 seconds to execute.Using job cluster:On at...
- 15453 Views
- 6 replies
- 15 kudos
- 2751 Views
- 2 replies
- 1 kudos
Skip Task Without Spinning Up Cluster
I have a Job Workflow with multiple sequential tasks executing R or Python scripts. Currently, we can skip one of these tasks (if it has already been run) by passing a parameter and skipping via the script. This requires a full spin up of a compute r...
- 2751 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Dave Wilson Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 1 kudos
- 4819 Views
- 1 replies
- 3 kudos
Resolved! Set Workflow Job Concurrency Limit
Hi Everyone,I need a job to be triggered every 5 minutes. However, if that job is already running, it must not be triggered again until that run is finished. Hence, I need to set the maximum run concurrency for that job to only one instance at a time...
- 4819 Views
- 1 replies
- 3 kudos
- 3 kudos
@Michael Okulik :To ensure that a Databricks job is not triggered again until a running instance of the job is completed, you can set the maximum concurrency for the job to 1. Here's how you can configure this in Databricks:Go to the Databricks work...
- 3 kudos
- 3323 Views
- 2 replies
- 2 kudos
Resolved! Workflow triggered by CLI shown "manually" triggered
We trigger different Worflows by ADF.These workflows will be shown triggered "manually".Is this behaviour intentional? At least for users, this is confusing.ADF-triggered Run: Databricks-Workflows:
- 3323 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @J. G., Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does. Your feedback will...
- 2 kudos
- 5608 Views
- 3 replies
- 3 kudos
Resolved! Impacts of running multiple jobs in parallel that refers the same notebook
Can I run multiple jobs(for example: 100+) in parallel that refers the same notebook? I supply each job with a different parameter. If we can do this, what would be the impact? (for example: reliability, performance, troubleshooting etc. )Example: N...
- 5608 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Murthy Ramalingam Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 3 kudos
- 873 Views
- 0 replies
- 0 kudos
Return branch name as variable when running workflow from Repo
I have a way to pull this when running from a notebook, but my teams runs workflows remotely from a repo. Is there a way to get the branch name when running from the repo?
- 873 Views
- 0 replies
- 0 kudos