Databricks Community

SimonY · 03-25-2022

Hello,I ran a spark stream job to ingest data from kafka to test Trigger.AvailableNow.What's environment the job run ?1: Databricks runtime 10.32: Azure cloud3: 1 Driver node + 3 work nodes( 14GB, 4core)val maxOffsetsPerTrigger = "500"spark.conf.set...

SimonY · 03-28-2022

@Karli Watsica , thanks for help. This issue has been fixed in databricks 10.4 and spark 3.3.[SPARK-36649] [SQL] Support Trigger.AvailableNow on Kafka data sourcehttps://docs.databricks.com/release-notes/runtime/10.4.html

Databricks Community

User Stats

User Activity

Trigger.AvailableNow does not support maxOffsetsPerTrigger in Databricks runtime 10.3

Re: Trigger.AvailableNow does not support maxOffsetsPerTrigger in Databricks runtime 10.3