cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
KKo
by • Contributor III
  • 8852 Views
  • 5 replies
  • 0 kudos

Move whole workflow from Dev to Prod

I have a workflow created in Dev, now I want to move the whole thing to prod and schedule it. The workflow has multiple notebooks, dependent libraries, parameters and such. How to move the whole thing to prod, instead of moving each notebooks and rec...

Data Engineering
Reply
  • 8852 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
mkassa
New Contributor II
  • 0 kudos

I ended up creating a python script to just do the export, here is the code below.It will match on Job name, if it matches it will update otherwise it will import. import requests source_token = '' source_instance = 'adb-000000000000000.00.azuredata...

  • 0 kudos
4 More Replies
joao_vnb
by • New Contributor III
  • 58761 Views
  • 7 replies
  • 11 kudos

Resolved! Automate the Databricks workflow deployment

Hi everyone,Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?

Data Engineering
Reply
  • 58761 Views
  • 7 replies
  • 11 kudos
View Replies
Latest Reply
asingamaneni
New Contributor II
  • 11 kudos

Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflowYou can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic w...

  • 11 kudos
6 More Replies
xneg
by • Contributor
  • 14184 Views
  • 12 replies
  • 9 kudos

PyPI library sometimes doesn't install during workflow execution

I have a workflow that is running upon a job cluster and contains a task that requires prophet library from PyPI:{ "task_key": "my_task", "depends_on": [ { "task_key": "<...>...

Data Engineering
Reply
  • 14184 Views
  • 12 replies
  • 9 kudos
View Replies
Latest Reply
Vartika
Databricks Employee
  • 9 kudos

Hey @Eugene Bikkinin​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...

  • 9 kudos
11 More Replies
giuseppegrieco
by • New Contributor III
  • 17123 Views
  • 5 replies
  • 6 kudos

Workflow service principle owned can't checkout git repository

I am trying to deploy a workflow where the owner is a service principal and I am using git integration (backend with azure devops), when I run the workflow it says that it doesn't have permissions to checkout the repo.run failed with error message F...

Data Engineering
Reply
  • 17123 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Giuseppe Grieco​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us ...

  • 6 kudos
4 More Replies
jonathan-dufaul
by • Valued Contributor
  • 10503 Views
  • 2 replies
  • 0 kudos

Resolved! Error updating workflow, webhook not found?

I have no idea what this error means or what it could mean. When I'm trying to save a workflow I get a popup saying this:

image
Data Engineering
Reply
  • 10503 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Robin_LOCHE
New Contributor II
  • 0 kudos

I had the same issue, thanks for the info! Apparently it's also possible to fix it by removing all the actual notification in the interface (the bugged one is not displayed, but if you remove everything for some reason it removes the bugged one too)....

  • 0 kudos
1 More Replies
Ria
by • New Contributor
  • 2972 Views
  • 4 replies
  • 0 kudos

How to build master workflow for all the jobs present in workflow using databricks?

Suppose there are multiple job have been created using databricks workflow, now the requirement is to make one master workflow to trigger all the workflow depending on different condition like: some are supposed to trigger on daily basis, some on mon...

Data Engineering
Reply
  • 2972 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
pvignesh92
Honored Contributor
  • 0 kudos

@Ria Hi , This feature was in development when I attended last Quarter Roadmap and I thought it is available in the latest versions or could be even in Private Preview. You can check with your Databricks Solution Architect. Even if not now, could be ...

  • 0 kudos
3 More Replies
apiury
by • New Contributor III
  • 5932 Views
  • 9 replies
  • 14 kudos

Resolved! Pipeline workflow dude

Hi! I have a problem. I'm using an autoloader to ingest data from raw to a Delta Lake, but when my pipeline starts, I want to apply the pipeline only to the new data. The autoloader ingests data into the Delta Lake, but now, how can I distinguish the...

Data Engineering
Reply
  • 5932 Views
  • 9 replies
  • 14 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 14 kudos

Hi @Alejandro Piury Pinzón​ We haven't heard from you since the last response from @Tyler Retzlaff​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be he...

  • 14 kudos
8 More Replies
timothy_uk
by • New Contributor III
  • 1117 Views
  • 1 replies
  • 1 kudos

Mysterious simultaneous long-running Databricks Workflows

Hi,This happened across 4x seemingly unrelated workflows at the same time of the day - all 4x workflows eventually completed successfully. It appeared that all workflows sat idling despite triggering via the Jobs API. The two symptoms I have observed...

Data Engineering
Reply
  • 1117 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Timothy Lin​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
AmanSehgal
by • Honored Contributor III
  • 18801 Views
  • 6 replies
  • 15 kudos

Job cluster vs All purpose cluster

Environment: AzureI've a workflow that takes approximately a minute to execute and I want to run the job every 2 minutes.. All purpose cluster:On attaching all purpose cluster to the job, it takes approx. 60 seconds to execute.Using job cluster:On at...

Data Engineering
Reply
  • 18801 Views
  • 6 replies
  • 15 kudos
View Replies
Latest Reply
Priyag1
Honored Contributor II
  • 15 kudos

Thanks for sharing

  • 15 kudos
5 More Replies
Dave_Nithio
by • Contributor
  • 3189 Views
  • 2 replies
  • 1 kudos

Skip Task Without Spinning Up Cluster

I have a Job Workflow with multiple sequential tasks executing R or Python scripts. Currently, we can skip one of these tasks (if it has already been run) by passing a parameter and skipping via the script. This requires a full spin up of a compute r...

Data Engineering
Reply
  • 3189 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Dave Wilson​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 1 kudos
1 More Replies
sage5616
by • Valued Contributor
  • 6018 Views
  • 1 replies
  • 3 kudos

Resolved! Set Workflow Job Concurrency Limit

Hi Everyone,I need a job to be triggered every 5 minutes. However, if that job is already running, it must not be triggered again until that run is finished. Hence, I need to set the maximum run concurrency for that job to only one instance at a time...

Data Engineering
Reply
  • 6018 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@Michael Okulik​ :To ensure that a Databricks job is not triggered again until a running instance of the job is completed, you can set the maximum concurrency for the job to 1. Here's how you can configure this in Databricks:Go to the Databricks work...

  • 3 kudos
Jkb
by • New Contributor II
  • 3661 Views
  • 0 replies
  • 1 kudos

Workflow triggered by CLI shown "manually" triggered

We trigger different Worflows by ADF.These workflows will be shown triggered "manually".Is this behaviour intentional? At least for users, this is confusing.ADF-triggered Run: Databricks-Workflows: 

ADF_Monitor manually1 manually2
Data Engineering
Reply
  • 3661 Views
  • 0 replies
  • 1 kudos
Murthy1
by • Contributor II
  • 6517 Views
  • 3 replies
  • 3 kudos

Resolved! Impacts of running multiple jobs in parallel that refers the same notebook

Can I run multiple jobs(for example: 100+) in parallel that refers the same notebook? I supply each job with a different parameter. If we can do this, what would be the impact? (for example: reliability, performance, troubleshooting etc. )Example: N...

Data Engineering
Reply
  • 6517 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Murthy Ramalingam​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 3 kudos
2 More Replies
Labels