topic Why saving pyspark df always converting string field to number? in Get Started Discussions

Why saving pyspark df always converting string field to number?

georgeyjy — Tue, 21 May 2024 07:24:47 GMT

import pandas as pd from pyspark.sql.types import StringType, IntegerType from pyspark.sql.functions import col save_path = os.path.join(base_path, stg_dir, "testCsvEncoding") d = [{"code": "00034321"}, {"code": "55964445226"}] df = pd.DataFrame(d) spark_df = spark.createDataFrame(df) spark_df.display() field = "code" spark_df = spark_df.withColumn(field, col(field).cast(StringType())) spark_df.display() spark_df.coalesce(1).write.mode('overwrite').format("csv").option("header", "true").option("encoding", "gbk").save(save_path)

example codes are as above, i am trying to convert pandas df to pyspark df, and then save data to a csv file, but somehow it always convert a string field to a number field in csv file as attached. Anyone can help?

Re: Why saving pyspark df always converting string field to number?

daniel_sahal — Tue, 21 May 2024 07:38:42 GMT

@georgeyjy
Try opening the CSV as text editor. I bet that Excel is automatically trying to detect the schema of CSV thus it thinks that it's an integer.

Re: Why saving pyspark df always converting string field to number?

georgeyjy — Tue, 21 May 2024 08:36:20 GMT

Oh, that's it, i tried to open it in text editor and the result is correct.

Many thanks!