Comment
Valued Contributor II

Great job, thanks for sharing!

Some small hints from my end:

  • if You also care about the latency of processing bronze to silver using structured streaming setting the parameter maxFilesPerTrigger will be important.
  • start with small cluster setup (I've used DS3_v2 single node for kafka to bronze for each topic using kafka connector), if needed scale vertically.
  • consider setting the retries to -1, jobs will fail once in a while and You propably want them to restart automatically.

Best!