I am processing the daily data from checkpoint to checkpoint everyday by using for each batch in streaming way.df.writeStream.format("delta")
.option("checkpointLocation", "dbfs/loc")
.foreachBatch(transform_and_upsert)
.outpu...
I have tried both,What I am triggering is a job, in job first I set within notebook level by adding spark.conf.setThen I also added some configs in job cluster
