I have a use case where my file has data in Thai characters. The source location is azure blob storage, here files are stored in text format. I am using the following code to read the file, but when I am downloading the data from catalog it encloses data in quotes which I don't want.

input_df = (
            spark.read.format("text")
            .option("ignoreLeadingWhiteSpace", "false")
            .option("ignoreTrailingWhiteSpace", "false")
            .option("encoding", encoding)
            .option("keepUndefinedRows", True)
            .load(file_path)
            .withColumn("decoded_text", expr(f"regexp_replace(decode(value, '{encoding}'), '^\"|\"$', '')"))
            .drop("value")
            .withColumnRenamed("decoded_text", "value")
        )