topic Re: pyspark - regexp_extract in Data Engineering

pyspark - regexp_extract

weldermartins — Wed, 23 Nov 2022 17:41:29 GMT

hello everyone, I'm creating a regex expression to fetch only the value of a string, but some values are negative. I am not able to create the rule to compose the negative value. can you help me?

from pyspark.sql.functions  import regexp_extract
from pyspark.sql.types import StructType,StructField, StringType
 
data = [("01","[$R$-pt-BR] 150.00"),
        ("02", "-[$R$-pt-BR] 379.52" ),
        ("03", "[$R$-pt-BR] 185.16" ),
        ("04", "[$R$-pt-BR] 185.16" ),]
 
schema = StructType([ \
    StructField("id",StringType(),True), \
    StructField("description",StringType(),True), 
  ])
 
df = spark.createDataFrame(data=data,schema=schema)
df.display()
 
df1 = df\
.withColumn("value", regexp_extract('description', r"[\d]{1,4}.[\d]{1,4}", 0))
df1.display()

Re: pyspark - regexp_extract

weldermartins — Wed, 23 Nov 2022 18:50:17 GMT

@Werner Stinckens

can you help me?

Re: pyspark - regexp_extract

NhatHoang — Thu, 24 Nov 2022 08:59:04 GMT

Hi there,

Create a column to catch the minus "-": pattern is: "^[\-]?"
Create a column to catch the digits that you already done.
Concat these two columns above.

Hope it fit your requirement. 🙂

Re: pyspark - regexp_extract

weldermartins — Thu, 24 Nov 2022 12:26:51 GMT

I found another solution, but I didn't want to give up on regex. If you find a way, be sure to post. Thanks.

Re: pyspark - regexp_extract

weldermartins — Thu, 24 Nov 2022 13:46:41 GMT

df1 = df\
.withColumn("value", regexp_extract('description', "[\d]{1,4}.[\d]{1,4}", 0))\
.withColumn("operador", regexp_extract('description', "^[\-]?", 0))\
.withColumn("value2", concat("operador","value"))
df1.display()

@Nhat Hoang , Thanks.

Re: pyspark - regexp_extract

-werners- — Thu, 24 Nov 2022 15:36:04 GMT

\s*\[[^)]*\] removes the square brackets and everything inside it and the space too (well actually substitutes it with nothing).

https://regex101.com/r/tv9pbJ/1

Haven't checked if spark can do regex substitution.

Re: pyspark - regexp_extract

mcwir — Thu, 01 Dec 2022 11:25:27 GMT

its like you need to find this pattern : "^[\-]?"

Re: pyspark - regexp_extract

ErinArmistead — Thu, 16 Mar 2023 12:50:31 GMT

Have you found the answer? If you are a student in college or school searching for free essay examples online, you may want to visit the website https://writinguniverse.com/free-essay-examples/soccer/ here you will find a vast collection of free essay examples related to various topics, including soccer. These essay examples can be valuable resources to help you complete your essay assignments.