Just to update the post, this does work:

  • You have a timestamp column
  • Generate a date column from the timestamp column
  • Partition on that generated date column
  • Write a query that filters on the original timestamp column
    • Databricks will only scan partitions within the date range.