Thanks Raphael! That's helpful. I'll look into the links.If I could ask you one more question, do you have any references or links to how upscaling and downscaling of the number of workers and cluster size is done for structured streaming? I have a u...
Hi @raphaelblg , thanks a lot for providing an elaborate answer. Do you happen to you, by any chance, of some solutions that developers use to track a consumer lag when streaming with Spark from a Kafka topic? It's a rather essential knowledge to hav...
Could you please provide an example of the proposed workaround? Nothing what I have tried helps, there is always the same error as in the original post. This is very frustrating to say the least - the inability to switch properly and natively to a lo...
