Reading two big tables within each forEachBatch processing method
I am reading changes from the cdf with availableOnce=True, processing data from checkpoint to checkpoint. During each batch, I perform transformations, but I also need to read two large tables and one small table. Does Spark read these tables from sc...
- 6 Views
- 0 replies
- 0 kudos