jb1z
Contributor

The problem was fixed by this import

from pyspark.sql import functions as F
 
then using F.lit() instead of F.col
.withColumn('ingestion_date', F.lit(folder_date))
 
Sorry code formatting is not working at the moment.

View solution in original post