https://community.databricks.com/t5/data-engineering/data-ingestion-issue-with-thai-data/m-p/108053#M42963 <P>I have a use case where my file has data in Thai characters. The source location is azure blob storage, here files are stored in text format. I am using the following code to read the file, but when I am downloading the data from catalog it encloses data in quotes which I don't want.</P><P>&nbsp;</P><LI-CODE lang="python">input_df = ( spark.read.format("text") .option("ignoreLeadingWhiteSpace", "false") .option("ignoreTrailingWhiteSpace", "false") .option("encoding", encoding) .option("keepUndefinedRows", True) .load(file_path) .withColumn("decoded_text", expr(f"regexp_replace(decode(value, '{encoding}'), '^\"|\"$', '')")) .drop("value") .withColumnRenamed("decoded_text", "value") )</LI-CODE> Fri, 31 Jan 2025 11:15:57 GMT hardeeksharma 2025-01-31T11:15:57Z https://community.databricks.com/t5/data-engineering/data-ingestion-issue-with-thai-data/m-p/108053#M42963 <P>I have a use case where my file has data in Thai characters. The source location is azure blob storage, here files are stored in text format. I am using the following code to read the file, but when I am downloading the data from catalog it encloses data in quotes which I don't want.</P><P>&nbsp;</P><LI-CODE lang="python">input_df = ( spark.read.format("text") .option("ignoreLeadingWhiteSpace", "false") .option("ignoreTrailingWhiteSpace", "false") .option("encoding", encoding) .option("keepUndefinedRows", True) .load(file_path) .withColumn("decoded_text", expr(f"regexp_replace(decode(value, '{encoding}'), '^\"|\"$', '')")) .drop("value") .withColumnRenamed("decoded_text", "value") )</LI-CODE> Fri, 31 Jan 2025 11:15:57 GMT https://community.databricks.com/t5/data-engineering/data-ingestion-issue-with-thai-data/m-p/108053#M42963 hardeeksharma 2025-01-31T11:15:57Z https://community.databricks.com/t5/data-engineering/data-ingestion-issue-with-thai-data/m-p/108177#M42989 <P>Do the quotes exist in original data?</P> Fri, 31 Jan 2025 18:48:16 GMT https://community.databricks.com/t5/data-engineering/data-ingestion-issue-with-thai-data/m-p/108177#M42989 Lakshay 2025-01-31T18:48:16Z