cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Mohit_m
by • Valued Contributor II
  • 21136 Views
  • 3 replies
  • 4 kudos

Resolved! How to get the Job ID and Run ID and save into a database

We are having Databricks Job running with main class and JAR file in it. Our JAR file code base is in Scala. Now, when our job starts running, we need to log Job ID and Run ID into the database for future purpose. How can we achieve this?

Data Engineering
Reply
  • 21136 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Bruno-Castro
New Contributor II
  • 4 kudos

That article is for members only, can we also specify here how to do it (for those that are not medium members?). Thanks!

  • 4 kudos
2 More Replies
cmilligan
by • Contributor II
  • 2889 Views
  • 3 replies
  • 2 kudos

Dropdown for parameters in a job

I want to be able to denote the type of run from a predetermined list of values that a user can choose from when kicking off a run using different parameters. Our team does standardized job runs on a weekly cadence but can have timeframes that change...

Data Engineering
Reply
  • 2889 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
dev56
New Contributor II
  • 2 kudos

Hi @cmilligan , I have a similar requirement and would really be grateful if you could provide me with any information on how to fix this issue. Thanks a lot!

  • 2 kudos
2 More Replies
Dave_Nithio
by • Contributor
  • 2785 Views
  • 2 replies
  • 1 kudos

Skip Task Without Spinning Up Cluster

I have a Job Workflow with multiple sequential tasks executing R or Python scripts. Currently, we can skip one of these tasks (if it has already been run) by passing a parameter and skipping via the script. This requires a full spin up of a compute r...

Data Engineering
Reply
  • 2785 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Dave Wilson​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 1 kudos
1 More Replies
cmilligan
by • Contributor II
  • 1685 Views
  • 2 replies
  • 0 kudos

Delay in retrieving task variables from a notebook utilizing %run

I have a databricks workflow where the first task is to set up task parameters for other notebooks to use later in the process. Since these are variables that are used in all of my notebooks, I have opted to assign them in a shared notebook and call ...

Data Engineering
Reply
  • 1685 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Please refer: https://docs.databricks.com/notebooks/notebook-workflows.htmlCould you please provide the error received?

  • 0 kudos
1 More Replies
ACK
by • New Contributor II
  • 2900 Views
  • 2 replies
  • 2 kudos

Resolved! How do I pass kwargs to wheel method?

Hi,I have a method named main it takes **kwargs as a parameter. def main(**kwargs): parameterOne = kwargs["param-one"] parameterTwo = kwargs["param-two"] parameterThree = kwargs["param-optional-one"] if "param-optional-one" in kwargs else...

Data Engineering
Reply
  • 2900 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

it is command-line parameters so it is like ---param-one=testyou can test it with ArgumentParserfrom argparse import ArgumentParser   parser = ArgumentParser() parser.add_argument("--param-one", dest="parameterOne")   args = parser.parse_args()

  • 2 kudos
1 More Replies
Labels