Lakeflow Pipelines Trying to Read accented file with spark.readStream but failure

AmarKap
New Contributor

‎10-20-2025 11:05 AM

Trying to read a accented file(French characters) but the spark.readStream function is not working and special characters turn into something strange(ex. �)

 
            spark.readStream
            .format("cloudfiles")
            .option("cloudFiles.format", "text")
            .option("encoding", "ISO-8859-1")

Tried both ISO-8859-1 and UTF-8. 
Tried with and without  .option("cloudFiles.format", "text")
Files do not contains .txt extension

 

Labels:
0 Kudos

K_Anudeep
Databricks Employee
Databricks Employee

‎10-21-2025 12:38 AM

Hello @AmarKap ,

When Spark decodes CP1252 bytes as UTF-8/ISO-8859-1, you’ll see the replacement char like �

Can you read the file as :

df = (spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "text")
.option("encoding", "windows-1252") # or "CP1252"
.load("s3://.../path")) 

Anudeep