topic New to PySpark in Get Started Discussions

New to PySpark

Anku_ — Fri, 22 Mar 2024 11:08:09 GMT

Hi all,

I am trying to get the domain from an email field using below expression; but getting an error.

Kindly help.

df.select(df.email, substring(df.email,instr(df.email,'@'),length(df.email).alias('domain')))

Re: New to PySpark

Anku_ — Fri, 22 Mar 2024 11:18:39 GMT

Note: I have also tried:

df.select(df.email, substring(df.email,instr(df.email,'@')+1,length(df.email) - instr(df.email,'@')+1).alias('domain'))

Re: New to PySpark

Walter_C — Sat, 23 Mar 2024 16:09:39 GMT

In your case, you want to extract the domain from the email, which starts from the position just after '@'. So, you should add 1 to the position of '@'. Also, the length of the substring should be the difference between the total length of the email and the position of '@'.

Can you try with the below:

from pyspark.sql.functions import instr, length, substring

df.select(df.email, substring(df.email, instr(df.email, '@') + 1, length(df.email) - instr(df.email, '@')).alias('domain'))

This code will create a new column 'domain' that contains the domain of the email.