cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

ADF pipeline fails when passing the parameter to databricks

g96g
New Contributor III

I have project where I have to read the data from NETSUITE using API. Databricks Notebook runs perfectly when I manually insert the table names I want to read from the source. I have dataset (csv) file in adf with all the table names that I need to read but some of the tables fail. When I run notebook for those failed tables it gives me an output and writing it azure datalake with no problem. Yesterday I had same issue and tryied to uplaod the new CSV ( with table names) in azure dataset and worked. But this morning it gives me again the same error.

Error code 3204

1 ACCEPTED SOLUTION

Accepted Solutions

g96g
New Contributor III

adls gen2, but I think I fixed the probelm but reducing the amount in the ForEach activity. Previoulsy I had batch count 10

image

View solution in original post

8 REPLIES 8

-werners-
Esteemed Contributor III

I think we gonna need some more detail on this.

How do you fetch the values from ADF in Databricks? Using widgets, system variables, spark config?

You mention a csv in ADF, how do you process it? And what if this csv is empty?

g96g
New Contributor III

I use a widgets. CSV is not empty, I can see data in preview. and also in the output too when I debug

-werners-
Esteemed Contributor III

are you sure the csv does not contain special characters etc?

csv files are known for shenanigans like that.

Passing variables from ADF to Databricks itself works fine, many people use it.

So my guess is there is something fishy in the values themselves so Spark cannot read them or the table does not exist (because of invalid characters etc).

g96g
New Contributor III

if it worked yesterday why it should not work today? the csv is static does not changes. Also, I tried to put the print statement to see if it prints the table name from csv and it does. Moreover, when I run the databricks manually it works perfectly, no error. I get my desired putput so this does not makes sense why its failing. Yesterday I had similar problem, my table names were written some upper case som lower, i changed them all to lower case, uploaded new file and it run perfectly 0 errors. then set the schedule and this morining it failed.

-werners-
Esteemed Contributor III

where did you save the csv file? You mention an Azure dataset, but can you be more specific? A blob storage, adls gen2?

g96g
New Contributor III

adls gen2, but I think I fixed the probelm but reducing the amount in the ForEach activity. Previoulsy I had batch count 10

image

mcwir
Contributor

Have you tried do debug the json payload of adf trigger ? maybe it wrongly conveys tables names

g96g
New Contributor III

i fixed it you can see my reply in the above comment

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group