SQL expr undefined function 'LEN'

PhillT — Fri, 28 Jul 2023 18:32:22 GMT

Getting this error message on our production cluster when I run a notebook that uses the SQL expr function that call the LEN() funciton

example code:

df = df.withColumn("POL", expr("CASE WHEN SRC_SYSTEM = 'X' THEN CONCAT('08' , SUBSTRING(POL, 3, LEN(POL) -2)) ELSE POL END"))

Below is the error message. The notebook runs with out issue on my machine on our dev cluster, issue happend on our prod cluster. Any one has any ideas?

Exception happened for incremental load. Exception-> Undefined function: 'LEN'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 87.

I have the following module import...

from pyspark.sql.functions import *

from pyspark.sql.types import *

from pyspark.sql import functions as F, Window

from pyspark.sql.functions import col, create_map, lit, expr

from itertools import chain

from pyspark import StorageLevel

from pyspark.sql.functions import coalesce

Re: SQL expr undefined function 'LEN'

daniel_sahal — Mon, 31 Jul 2023 05:05:49 GMT

@PhillT
There's no "LEN" function. You should use "LENGTH" instead.

https://spark.apache.org/docs/2.3.0/api/sql/index.html#length

topic Re: SQL expr undefined function 'LEN' in Data Engineering

SQL expr undefined function 'LEN'

Re: SQL expr undefined function 'LEN'