ThomazRossito
Contributor

‎04-14-2024 05:54 AM

Hi,

I noticed that in the scheme you are creating, there are more columns than in your csv file, I was able to understand that the final result needs to include the 6 columns

I would use withColumn, for the 3 columns that do not exist in the file

Below is an example

 

df = (
spark.read
.format("csv")
.option("header","true")
.option("delimiter","|")
.load("pathFile")
.withColumn("TRANSACTION", lit('null').cast(StringType()))
.withColumn("FROM", lit('null').cast(StringType()))
.withColumn("TO", lit('null').cast(StringType()))
)
 
Hope this helps
Att.
Thomaz Antonio Rossito Neto
Master Data Specialist - Data Architect | Data Engineer @ CI&T
0 Kudos