cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

arthur_wang
by New Contributor
  • 3827 Views
  • 2 replies
  • 1 kudos

How does Task Orchestration compare to Airflow (for Databricks-only jobs)?

One of my clients has been orchestration Databricks notebooks using Airflow + REST API. They're curious about the pros/cons of switching these jobs to Databricks jobs with Task Orchestration.I know there are all sorts of considerations - for example,...

  • 3827 Views
  • 2 replies
  • 1 kudos
Latest Reply
Shourya
New Contributor III
  • 1 kudos

@Kaniz Fatma​ Hello Kaniz, I'm currently working with a major Enterprise Client looking to make the choice between the Airflow vs Databricks for Jobs scheduling. Our Entire code base is in Databricks and we are trying to figure out the complexities t...

  • 1 kudos
1 More Replies
AlexandrePellet
by New Contributor
  • 3666 Views
  • 3 replies
  • 3 kudos

Resolved! Orchestration Preview - Error updating [JobName] - Either new_cluster or existing_cluster_id must be specified.

I've created a new job with the new UI / feature enabled. I managed to create one task with a new job cluster successfully but when adding a second task with a new job cluster and trying to save it I received the following error: Error updating [Job...

  • 3666 Views
  • 3 replies
  • 3 kudos
Latest Reply
eignerfr
New Contributor II
  • 3 kudos

Hi, I have a similar issue.I'm using the 14 day free trial, setting up the default basic-starter cluster.Then I just followed the basic introduction of dbx with python. https://docs.databricks.com/dev-tools/dbx.htmlCommand Execute is working, but whe...

  • 3 kudos
2 More Replies
Robbie
by New Contributor III
  • 3198 Views
  • 2 replies
  • 4 kudos

Resolved! Why can't I create new jobs? ("You are not entitled to run this type of task...")

This morning I encountered an issue when trying to create a new job using the Workflows UI (in browser). Never had this issue before.The error message that appears is:"You are not entitled to run this type of task, please contact your Databricks admi...

Screenshot including the error message
  • 3198 Views
  • 2 replies
  • 4 kudos
Latest Reply
Robbie
New Contributor III
  • 4 kudos

@Kaniz Fatma​ @Philip Nord​, thanks!I was able to do what I needed by cloning an existing job & modifying. It's fine as a temporary fix for now.Thanks again for the response-- good to know you're aware of it & this isn't anything on my end.

  • 4 kudos
1 More Replies
Mohit_m
by Valued Contributor II
  • 1239 Views
  • 1 replies
  • 4 kudos

Enabling of Task Orchestration feature in Jobs via API as well Databricks supports the ability to orchestrate multiple tasks within a job. You must en...

Enabling of Task Orchestration feature in Jobs via API as wellDatabricks supports the ability to orchestrate multiple tasks within a job. You must enable this feature in the admin console. Once enabled, this feature cannot be disabled. To enable orch...

  • 1239 Views
  • 1 replies
  • 4 kudos
Latest Reply
Prabakar
Databricks Employee
  • 4 kudos

@Mohit Miglani​ this will be really helpful for those who prefer CLI / API more than the UI.

  • 4 kudos
daniil_terentye
by New Contributor III
  • 1009 Views
  • 1 replies
  • 0 kudos

New task orchestration tool

Hi everybody! Is it possible to use one job cluster for multiple tasks using new task orchestration tool? If so then please tell how. If it's impossible then this new tool looks useless Regards, Daniil.

  • 1009 Views
  • 1 replies
  • 0 kudos
Latest Reply
daniil_terentye
New Contributor III
  • 0 kudos

Databricks team, are there any news about it?

  • 0 kudos
Labels