Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2025 05:03 AM
My job continue to running after is finished susccessfully this i my case, i enable useNotification
if self.autoloader_config.use_autoloader:
logger_file_ingestion.info("debut d'ecriture en mode streaming")
if self.write_mode.value.lower() == "append":
logger_file_ingestion.info("ecriture en mode %s", self.write_mode.value)
# Création de la configuration de base du stream
stream_writer = (df.writeStream
.format("delta")
.outputMode("append")
.option("checkpointLocation", self.checkpoint_location)
.option("mergeSchema", "true")
.trigger(availableNow=True))
# Ajout des partitions si nécessaire
if (self.source_name.lower() == "name") and (self.file_format.lower() == "parquet"😞
stream_writer = stream_writer.partitionBy("year", "day", "month")
elif (self.source_name.lower() == "test") and (self.file_format.lower() == "parquet"😞
stream_writer = stream_writer.partitionBy("day", "month", "year")
# Lancement du stream et capture de la référence
stream_writer.toTable("bronze")
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-04-2025 02:18 AM
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-08-2025 12:09 AM
Hi @MariuszK,
Thanks it works now. i have use awaitTermination()
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-08-2025 01:07 AM
Hi @seefoods ,
If it works, you can mark my answer as a solution so that if someone has the same problem, it will be easier to find an answer.