Thanks for your detailed response @Retired_mod! 

Regarding the reduce approach, it doesn't seem to work as outlined since reduce is not a member of org.apache.spark.sql.RelationalGroupedDataset

For the second approach, to clarify are you suggesting the following

1. Stream silver intermediate table writing with a MERGE INTO statement
2. Batch job to read output of (1) and write to gold writing with append
3. Stream gold table from (2) to power a different 3rd table