topic Re: ADF pipeline fails when passing the parameter to databricks in Data Engineering

ADF pipeline fails when passing the parameter to databricks

g96g — Thu, 01 Dec 2022 09:14:45 GMT

I have project where I have to read the data from NETSUITE using API. Databricks Notebook runs perfectly when I manually insert the table names I want to read from the source. I have dataset (csv) file in adf with all the table names that I need to read but some of the tables fail. When I run notebook for those failed tables it gives me an output and writing it azure datalake with no problem. Yesterday I had same issue and tryied to uplaod the new CSV ( with table names) in azure dataset and worked. But this morning it gives me again the same error.

Error code 3204

Re: ADF pipeline fails when passing the parameter to databricks

-werners- — Thu, 01 Dec 2022 09:30:37 GMT

I think we gonna need some more detail on this.

How do you fetch the values from ADF in Databricks? Using widgets, system variables, spark config?

You mention a csv in ADF, how do you process it? And what if this csv is empty?

Re: ADF pipeline fails when passing the parameter to databricks

g96g — Thu, 01 Dec 2022 09:34:18 GMT

I use a widgets. CSV is not empty, I can see data in preview. and also in the output too when I debug

Re: ADF pipeline fails when passing the parameter to databricks

-werners- — Thu, 01 Dec 2022 09:39:03 GMT

are you sure the csv does not contain special characters etc?

csv files are known for shenanigans like that.

Passing variables from ADF to Databricks itself works fine, many people use it.

So my guess is there is something fishy in the values themselves so Spark cannot read them or the table does not exist (because of invalid characters etc).

Re: ADF pipeline fails when passing the parameter to databricks

g96g — Thu, 01 Dec 2022 09:42:59 GMT

if it worked yesterday why it should not work today? the csv is static does not changes. Also, I tried to put the print statement to see if it prints the table name from csv and it does. Moreover, when I run the databricks manually it works perfectly, no error. I get my desired putput so this does not makes sense why its failing. Yesterday I had similar problem, my table names were written some upper case som lower, i changed them all to lower case, uploaded new file and it run perfectly 0 errors. then set the schedule and this morining it failed.

Re: ADF pipeline fails when passing the parameter to databricks

-werners- — Thu, 01 Dec 2022 09:54:22 GMT

where did you save the csv file? You mention an Azure dataset, but can you be more specific? A blob storage, adls gen2?

Re: ADF pipeline fails when passing the parameter to databricks

g96g — Thu, 01 Dec 2022 10:07:41 GMT

adls gen2, but I think I fixed the probelm but reducing the amount in the ForEach activity. Previoulsy I had batch count 10

Re: ADF pipeline fails when passing the parameter to databricks

mcwir — Thu, 01 Dec 2022 11:20:52 GMT

Have you tried do debug the json payload of adf trigger ? maybe it wrongly conveys tables names

Re: ADF pipeline fails when passing the parameter to databricks

g96g — Thu, 01 Dec 2022 12:45:50 GMT

i fixed it you can see my reply in the above comment