- 1509 Views
- 3 replies
- 3 kudos
Resolved! Trigger.AvailableNow does not support maxOffsetsPerTrigger in Databricks runtime 10.3
Hello,I ran a spark stream job to ingest data from kafka to test Trigger.AvailableNow.What's environment the job run ?1: Databricks runtime 10.32: Azure cloud3: 1 Driver node + 3 work nodes( 14GB, 4core)val maxOffsetsPerTrigger = "500"spark.conf.set...
- 1509 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
You'd be better off with 1 node with 12 cores than 3 nodes with 4 each. You're shuffles are going to be much better one 1 machine.
- 3 kudos