cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
User16790091296
by • Contributor II
  • 3142 Views
  • 1 replies
  • 0 kudos

How to create a databricks job with parameters via CLI?

I'm creating a new job in databricks using the databricks-cli:databricks jobs create --json-file ./deploy/databricks/config/job.config.jsonWith the following json:{ "name": "Job Name", "new_cluster": { "spark_version": "4.1.x-scala2.1...

Data Engineering
Reply
  • 3142 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
matthew_m
Databricks Employee
  • 0 kudos

This is an old post but still relevant for future readers, so will answer how it is done. You need to add base_parameters flag in the notebook_task config, like the following.   "notebook_task": { "notebook_path": "...", "base_parameters": { ...

  • 0 kudos
shan_chandra
by • Databricks Employee
  • 2370 Views
  • 1 replies
  • 1 kudos

Resolved! Adding spark_conf tag on Jobs API

using Jobs API, when we create a new job to run on an interactive cluster, can we add spark_conf tag and specify spark config tuning parameters?

Data Engineering
Reply
  • 2370 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
shan_chandra
Databricks Employee
  • 1 kudos

 spark_conf needs to be set prior to the start of the cluster or have to restart the existing cluster. Hence, the spark_conf tag is available only on the job_cluster. you may have to set the configs manually on the interactive cluster prior to using ...

  • 1 kudos
F29
by • New Contributor
  • 2239 Views
  • 3 replies
  • 0 kudos

is possible duplicated a new job in another stage Databricks? Devops or some way?

I need duplicated a new job create in stage A in another stage, automatically. is posible?

Data Engineering
Reply
  • 2239 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

you may try to get the job details from our job api https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsGet and get the response to duplicate it.,

  • 0 kudos
2 More Replies
Robbie
by • New Contributor III
  • 2896 Views
  • 2 replies
  • 4 kudos

Resolved! Why can't I create new jobs? ("You are not entitled to run this type of task...")

This morning I encountered an issue when trying to create a new job using the Workflows UI (in browser). Never had this issue before.The error message that appears is:"You are not entitled to run this type of task, please contact your Databricks admi...

Screenshot including the error message
Data Engineering
Reply
  • 2896 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
Robbie
New Contributor III
  • 4 kudos

@Kaniz Fatma​ @Philip Nord​, thanks!I was able to do what I needed by cloning an existing job & modifying. It's fine as a temporary fix for now.Thanks again for the response-- good to know you're aware of it & this isn't anything on my end.

  • 4 kudos
1 More Replies
Labels