@Christine Pedersen​ :

You can achieve this without collecting data into a list using Spark's built-in DataFrame operations.

You can use the join operation to filter DF2 based on the distinct values in the column from pysparkDF . Here's an example:

filtered_table = DF2.join(
    pysparkDF.select(<column_name>).distinct(),
    on=DF2.<column_name> == pysparkDF.<column_name>,
    how='inner'
)

This code will perform an inner join on DF2 and pysparkDF using the column name, which will effectively filter DF2 based on the distinct values of that column in pysparkDF. Note that this approach will return a new DataFrame rather than a list, which should be more efficient for larger datasets