So I'm querying data from parquet files that have a couple of billions records (table 1 or t1), and then have to filter and then join with other parquet files with another couple of billions records (t2).

This takes quite a long time to run (like 10hs or so for each query), and I'm seeing that after saving the results of filtering t1 into a temp view, every time I run a query using the results from the temp view, it scans the parquet files again and filters again.

I ended up creating a table in the databricks dbfs and inserting the results of the filtering of t1 and then querying that table for the joins, which runs much faster. So I'm wondering, does the temp view only saves the query but actually runs it every time it's used? Is there a more efficient way to do this without inserting into a table?