batch process autoloader

seefoods
Valued Contributor

My job continue to running after is finished susccessfully this i my case, i enable useNotification 

if self.autoloader_config.use_autoloader:
logger_file_ingestion.info("debut d'ecriture en mode streaming")

if self.write_mode.value.lower() == "append":
logger_file_ingestion.info("ecriture en mode %s", self.write_mode.value)

# Création de la configuration de base du stream
stream_writer = (df.writeStream
.format("delta")
.outputMode("append")
.option("checkpointLocation", self.checkpoint_location)
.option("mergeSchema", "true")
.trigger(availableNow=True))

# Ajout des partitions si nécessaire
if (self.source_name.lower() == "name") and (self.file_format.lower() == "parquet"😞
stream_writer = stream_writer.partitionBy("year", "day", "month")
elif (self.source_name.lower() == "test") and (self.file_format.lower() == "parquet"😞
stream_writer = stream_writer.partitionBy("day", "month", "year")

# Lancement du stream et capture de la référence
stream_writer.toTable("bronze")

MariuszK
Valued Contributor III

Did you try use .awaitTermination() on stream writing?

View solution in original post

seefoods
Valued Contributor

Hi @MariuszK, 
Thanks it works now. i have use awaitTermination()

MariuszK
Valued Contributor III

Hi @seefoods ,
If it works, you can mark my answer as a solution so that if someone has the same problem, it will be easier to find an answer.

View solution in original post