@Sanjay Jain​ sorry missed one thing. .trigger(once=True)​ doesn't support rate limiters. You can use .trigger(availableNow=True)​ instead.

ref: https://docs.databricks.com/structured-streaming/triggers.html#configuring-incremental-batch-process...

spark.readStream.format("delta")

.option("maxFilesPerTrigger", 200)

.load(silver_path)

.writeStream

.option("checkpointLocation", gold_checkpoint_path)

.trigger(availableNow=True)

.foreachBatch(foreachBatchFunction)

.start()