ADF pipeline fails when passing the parameter to databricks

[g96g]

I have project where I have to read the data from NETSUITE using API. Databricks Notebook runs perfectly when I manually insert the table names I want to read from the source. I have dataset (csv) file in adf with all the table names that I need to read but some of the tables fail. When I run notebook for those failed tables it gives me an output and writing it azure datalake with no problem. Yesterday I had same issue and tryied to uplaod the new CSV ( with table names) in azure dataset and worked. But this morning it gives me again the same error.

Error code 3204

[-werners-]

I think we gonna need some more detail on this.

How do you fetch the values from ADF in Databricks? Using widgets, system variables, spark config?

You mention a csv in ADF, how do you process it? And what if this csv is empty?

[g96g]

I use a widgets. CSV is not empty, I can see data in preview. and also in the output too when I debug

[-werners-]

are you sure the csv does not contain special characters etc?

csv files are known for shenanigans like that.

Passing variables from ADF to Databricks itself works fine, many people use it.

So my guess is there is something fishy in the values themselves so Spark cannot read them or the table does not exist (because of invalid characters etc).

[g96g]

if it worked yesterday why it should not work today? the csv is static does not changes. Also, I tried to put the print statement to see if it prints the table name from csv and it does. Moreover, when I run the databricks manually it works perfectly, no error. I get my desired putput so this does not makes sense why its failing. Yesterday I had similar problem, my table names were written some upper case som lower, i changed them all to lower case, uploaded new file and it run perfectly 0 errors. then set the schedule and this morining it failed.

[-werners-]

where did you save the csv file? You mention an Azure dataset, but can you be more specific? A blob storage, adls gen2?

[g96g · Accepted solution]

adls gen2, but I think I fixed the probelm but reducing the amount in the ForEach activity. Previoulsy I had batch count 10

[mcwir]

Have you tried do debug the json payload of adf trigger ? maybe it wrongly conveys tables names

[g96g]

i fixed it you can see my reply in the above comment