Comment
11-22-2024
01:00 PM
- edited
11-22-2024
01:27 PM
11-22-2024
01:00 PM
- edited
11-22-2024
01:27 PM
Great job, thanks for sharing!
Some small hints from my end:
- if You also care about the latency of processing bronze to silver using structured streaming setting the parameter maxFilesPerTrigger will be important.
- start with small cluster setup (I've used DS3_v2 single node for kafka to bronze for each topic using kafka connector), if needed scale vertically.
- consider setting the retries to -1, jobs will fail once in a while and You propably want them to restart automatically.
Best!