Databricks Community

fabien_arnaud · 10-21-2024

I have a dataframe containing several columns among which 1 contains, for one specific record, just a comma, nothing else.When displaying the dataframe with the commanddisplay(df_input.where(col("erp_vendor_cd") == 'B6SA-VEN0008838')) The data is dis...

fabien_arnaud · 10-21-2024

I actually can't change the escape character because the double quote is the one being used by the source file and is required to correctly parse other columns in the dataframe such as the case below where the name column contains double quotes in th...

fabien_arnaud · 10-21-2024

Yes the dataframe reads from a CSV. Here is the code: df_input = (spark .read .format('CSV') .options(header= True, delimiter = ",", quot...

fabien_arnaud · 10-21-2024

Here is a screenshot of my code and the output:

Databricks Community

User Stats

User Activity

Data shifted when a pyspark dataframe column only contains a comma

Re: Data shifted when a pyspark dataframe column only contains a comma

Re: Data shifted when a pyspark dataframe column only contains a comma

Re: Data shifted when a pyspark dataframe column only contains a comma