topic Re: PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was 1 and the length of input was 2.'. in Data Engineering

PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was 1 and the length of input was 2.'.

Ancil — Tue, 17 Jan 2023 11:08:23 GMT

I have pandas_udf, its working for 1 rows, but I tried with more than one rows getting below error.

PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was 1 and the length of input was 2.'.

Code

@func.pandas_udf(StringType())
 
def find_data(inputs : Iterator[pd.Series]) -> Iterator[pd.Series]:
 
       for input in inputs :
 
           --doing logic have for loop, if etc.
 
           yield pd.Series(str(result_json))
 
 
 
df = df.withColumn("outData",find_data("inputData"))

Re: PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was 1 and the length of input was 2.'.

Hubert-Dudek — Tue, 17 Jan 2023 12:18:21 GMT

I was testing, and your function is correct. So you need to have an error in inputData

type (is all string) or with result_json. Please also check the runtime version. I was using 11 LTS.

Re: PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was 1 and the length of input was 2.'.

Ancil — Tue, 17 Jan 2023 12:41:41 GMT

Thanks @Hubert Dudek . Let me check the version.

Re: PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was 1 and the length of input was 2.'.

Ancil — Wed, 18 Jan 2023 05:20:20 GMT

Hi @Hubert Dudek

I tried, hard coded data Frame input data , its working as expected.

But if am loading same data from file getting above mentioned error, do you have any idea.