Partition In Spark with subqeury which include Union

[Jeewan]

I have a SQL query like this:
select ... from table1 where id in (slect id from table 1 where (some condition) UNION select id from table2 where (some condition)) table1

I have made a partition of 200 where upper bound is 200 and lower bound is 0 and partition will be done on partition_key column which has value ranged from 1 to 200. I am using JDBC connector. I am passing options("dbtable,table)  where table is the query mentioned above.

How will the internal query in Spark look like? Since we are using UNION will it not affect the partitioning? 

[Kaniz_Fatma]

Hi @Jeewan, When using a SQL query with a `UNION` in Spark, the process involves executing the subqueries within the `UNION`, combining their results, and then applying partitioning based on the specified `partition_key` column. First, Spark executes the individual subqueries and combines their results using `UNION` (which removes duplicates) or `UNION ALL` (which keeps duplicates). After this, Spark applies partitioning according to the `partition_key`, distributing the data across the specified partitions. The partitioning is applied to the combined result set, not directly influenced by the `UNION` operation itself.