@Kearon McNicol​ :

Yes, it's possible to remove duplicates in SQL. Here's an example query that achieves the same result as the Python code you provided:

SELECT DISTINCT *
FROM live.currStudents_ingest
WHERE file_modification_time IS NOT NULL;

In this SQL query, the DISTINCT keyword is used to remove duplicates from the result set. The WHERE clause filters out any rows where the file_modification_time column is NULL.

Note that this query assumes that live.currStudents_ingest is a Delta table in your Spark environment, and that you have access to run SQL queries against it.