topic Re: what api calls does autoloader make on s3? in Data Engineering

what api calls does autoloader make on s3?

SakuraDev1 — Wed, 16 Oct 2024 14:38:19 GMT

Hey guys I'm trying to find the estimate for an ingestion pipeline that uses autoloader on an S3 bucket every 2 minutes.

I found the pricing for s3 bucket api consumption but I am not certain what api calls will autoloader make.

Talking to chatGPT it told me i will have a List call per execution and then a get call per file per execution, but i am not certain of this answer

spark.readStream .format("cloudFiles") .option("cloudFiles.format", "json") .schema(json_schema_bronze) .load("s3://path") dfBronze.writeStream \ .format("delta") \ .option("checkpointLocation", checkpoint_dir_path_bronze) \ .outputMode("append") \ .trigger(processingTime="120 second") \ .table(bronze_table)

Re: what api calls does autoloader make on s3?

filipniziol — Wed, 16 Oct 2024 18:59:57 GMT

Hi @SakuraDev1 ,

LIST and GET make sense.

How autoloader works is monitors a specified location and then if the new file is discovered, it is being processed to bronze table.

So a LIST request is needed to check the files in the source directory, and then this list of files is compared against the list of already processed files. The unprocessed files are then loaded into bronze table. For every file you need a GET request to read the file and its content.

Re: what api calls does autoloader make on s3?

SakuraDev1 — Wed, 16 Oct 2024 20:11:09 GMT

Hey @filipniziol,

Thanks for the response, I've been trying to find some docs about this with no luck so far, do you happen to know any?