topic Re: autoloader running task batch in Data Engineering

autoloader running task batch

seefoods — Wed, 02 Jul 2025 09:51:25 GMT

Hello Guys,
I run task in batch mode with autoloader i enable option trigger (available now true). So, when my script finish, the continue running. Someone know who's happen?

Cordially ;

Re: autoloader running task batch

SP_6721 — Thu, 03 Jul 2025 12:56:26 GMT

Hi @seefoods ,

We usually use Trigger.AvailableNow when files arrive in batches rather than continuously. If your script keeps running even after processing, it could be that the job is still checking for any remaining files.

If it seems to be hanging for too long, try adding .awaitTermination()

Re: autoloader running task batch

seefoods — Thu, 03 Jul 2025 13:08:29 GMT

when i use .awaitTermination() it loop so long

Re: autoloader running task batch

seefoods — Thu, 03 Jul 2025 13:12:36 GMT

this is my script I enable this options when i read files on Volumes before write on delta table

(reader_stream.option("cloudFiles.format", self.file_format)
                     .option("cloudFiles.schemaLocation", self.schema_location)
                     .option("cloudFiles.useNotifications", True)
                     .option("cloudFiles.validateOptions", True)
                     .option("cloudFiles.schemaEvolutionMode", "rescue")
                     .option("cloudFiles.maxFilesPerTrigger", 1000))

f self.autoloader_config.use_autoloader:
    logger_file_ingestion.info("debut d'ecriture en mode streaming")

    if self.write_mode.value.lower() == "append":
        logger_file_ingestion.info("ecriture en mode %s", self.write_mode.value)

        # Création de la configuration de base du stream
        stream_writer = (df.writeStream
                         .format("delta")
                         .outputMode("append")
                         .option("checkpointLocation", self.checkpoint_location)
                         .option("mergeSchema", "true")
                         .trigger(availableNow=True))

        # Ajout des partitions si nécessaire
        if (self.source_name.lower() == "name") and (self.file_format.lower() == "parquet"😞
            stream_writer = stream_writer.partitionBy("year", "day", "month")
        elif (self.source_name.lower() == "test") and (self.file_format.lower() == "parquet"😞
            stream_writer = stream_writer.partitionBy("day", "month", "year")

        # Lancement du stream et capture de la référence
        stream_writer.toTable("bronze")