While writing stream data :

spark.table('table_name').writeStream works fine.

important settings are required for that:

A. Checkpointing :

1. Stores the current status of the streaming job with combination of "Write ahead log".

2. These checkpoint locations are unique.

B. Output Mode:

1. outputMode('append') : Only new records are added. default setting.

2. outputMode('completed') : Recalculated each time a write is triggered. Used mostly for aggregated tables.

3. outputMode('update') : Allows to perform complex upserts in streaming data. requires .start() command too.

C. Trigger Interval:

1. trigger(): if unspecified then, processingTime='500ms". Automatically detect and process all data in the source which are added post last run.

2. trigger(processingTime='2 minutes"): processes available data in micro batch at user specified time.

3. trigger(once=True): process all available data in one go (single batch).

4. trigger(availableNow=True): process all available data in multiple batches then stops. good option as long running steams can be bypassed with one fault tolerance guarantee.

D. .awaitTermination(): This option blocks execution of next cell until the incremental batch write has succeeded.