How to remove more than 4 byte characters using pyspark in databricks?

[eimis_pacheco]

Hi community,

We have the need of removing more than 4 byte characters using pyspark in databricks since these are not supported by amazon Redshift. Does someone know how can I accomplish this?

Thank you very much in advance

Regards

[Shalabh007]

assuming you are having a string type column in pyspark dataframe, one possible way could be

1. identify total number of characters for each value in column (say
2. identify no of bytes taken by each character (say b)
3. use substring() function to select first n characters where n = floor(4 / b)