Getting this error message on our production cluster when I run a notebook that uses the SQL expr function that call the LEN() funciton
example code:
df = df.withColumn("POL", expr("CASE WHEN SRC_SYSTEM = 'X' THEN CONCAT('08' , SUBSTRING(POL, 3, LEN(POL) -2)) ELSE POL END"))
Below is the error message. The notebook runs with out issue on my machine on our dev cluster, issue happend on our prod cluster. Any one has any ideas?
Exception happened for incremental load. Exception-> Undefined function: 'LEN'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 87.
I have the following module import...
from pyspark.sql.functions import *
from pyspark.sql.types import *
from pyspark.sql import functions as F, Window
from pyspark.sql.functions import col, create_map, lit, expr
from itertools import chain
from pyspark import StorageLevel
from pyspark.sql.functions import coalesce
