topic Re: How to pass job task parameters to another task in scala in Data Engineering

How to pass job task parameters to another task in scala

Sreekanth1 — Wed, 30 Nov 2022 18:56:42 GMT

Hi Team,

I have a requirement in workflow job. Job has two tasks, one is python-task and another one is scala-task (both are running their own cluster).

I have defined dbutils.job.taskValue in python which is not able to read value in scala because of job.taskValue is not supported in scala.

So, the question is,

how to pass custom parameters to task 2 ( scala )?.

Is dbutil.widget can work here by setText in python-task and read widget.get in scala-task ?.

Do we have any approaches?.

Note : We shouldn't use notebook.run.

UmaMahesh1 — Wed, 30 Nov 2022 19:10:34 GMT

Hi @Sreekanth Nallapa

Assuming that you want to pass python notebook params to scala nb. Else dbutils.jobs.taskValues is also available in scala.

If you want to pass parameters from py nb to scala nb or vice-versa,

try using dbutils.notebook.exit() and for the returnValue part, return the parameter you want.

You can then pull the exited value into your other notebook.

Cheers..

Ajay-Pandey — Tue, 06 Dec 2022 09:56:39 GMT

Hi @Sreekanth Nallapa please refer this link

This might help you in this