topic Re: Generating Spark SQL query using Python in Data Engineering

Generating Spark SQL query using Python

Constantine — Thu, 04 Nov 2021 18:36:30 GMT

I have a Spark SQL notebook on DB where I have a sql query like

SELECT * 
FROM table_name
WHERE 
     condition_1 = 'fname'  OR condition_1 = 'lname' OR condition_1 = 'mname' 
    AND condition_2 = 'apple' 
   AND condition_3 ='orange'

There are a lot of conditions, is there a way to define a python function and link it to sql to automatically generate the WHERE clause string

Re: Generating Spark SQL query using Python

Hubert-Dudek — Thu, 04 Nov 2021 20:39:38 GMT

fruits= ["apple", "orange"]
df.where(col("v").isin(fruits))

fruits= ["apple", "orange"]
sqlContext.sql("SELECT * FROM df WHERE v IN {0}".format(fruits))

Re: Generating Spark SQL query using Python

jose_gonzalez — Fri, 05 Nov 2021 00:16:45 GMT

Hi @John Constantine ,

I think you can also use arrays_overlap() for your OR statements docs here