topic Re: how to pass arguments and variables to databricks python activity from azure data factory in Data Engineering

how to pass arguments and variables to databricks python activity from azure data factory

MallikSunkara — Mon, 22 Jul 2019 14:45:12 GMT

how to pass arguments and variables to databricks python activity from azure data factory

Re: how to pass arguments and variables to databricks python activity from azure data factory

Anonymous — Mon, 22 Jul 2019 15:31:58 GMT

I do something like this...

In DBrix notebook:

dbutils.widgets.text("runDateYYYYMMDD", "")
runDate = dbutils.widgets.get("runDateYYYYMMDD")

In Data Factory:

Pipeline > Parameters tab > New Parameter:

Name: RunDate, Type:String, Default Value:

Data Bricks Notebook Activity > Settings tab > New Base Parameter:

Name=runDateYYYYMMDD, Value=pipeline().parameters.RunDate

Re: how to pass arguments and variables to databricks python activity from azure data factory

MallikSunkara — Tue, 23 Jul 2019 06:01:39 GMT

Hi Timkracht,

thanks for your reply.

is there any way to directly to read those parameters without using widgets. what we need is from azure data factory we are triggering python activity as python program file is stored in dbfs, now i need to access those variables in that python program.

what we need is which python files are being executed by which pipeline and activity of azure data factory.

thanks in advance.

Re: how to pass arguments and variables to databricks python activity from azure data factory

Anonymous — Tue, 23 Jul 2019 12:44:50 GMT

I'm afraid I do not have experience with that, just passing parameters through widgets in notebooks.

Re: how to pass arguments and variables to databricks python activity from azure data factory

CristianIspan — Thu, 27 May 2021 13:43:58 GMT

try importing argv from sys. Then if you have the parameter added correctly in DataFactory you could get it in your python script typing argv[1] (index 0 is the file path).