cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

what api calls does autoloader make on s3?

SakuraDev1
New Contributor II

Hey guys I'm trying to find the estimate for an ingestion pipeline that uses autoloader on an S3 bucket every 2 minutes.

I found the pricing for s3 bucket api consumption but I am not certain what api calls will autoloader make.

Talking to chatGPT it told me i will have a List call per execution and then a get call per file per execution, but i am not certain of this answer

spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .schema(json_schema_bronze)
    .load("s3://path")

dfBronze.writeStream \
    .format("delta") \
    .option("checkpointLocation", checkpoint_dir_path_bronze) \
    .outputMode("append") \
    .trigger(processingTime="120 second") \
    .table(bronze_table)

 

1 ACCEPTED SOLUTION

Accepted Solutions

filipniziol
Esteemed Contributor

Hi @SakuraDev1 ,

LIST and GET make sense.

How autoloader works is monitors a specified location and then if the new file is discovered, it is being processed to bronze table.

So a LIST request is needed to check the files in the source directory, and then this list of files is compared against the list of already processed files. The unprocessed files are then loaded into bronze table. For every file you need a GET request to read the file and its content.

View solution in original post

2 REPLIES 2

filipniziol
Esteemed Contributor

Hi @SakuraDev1 ,

LIST and GET make sense.

How autoloader works is monitors a specified location and then if the new file is discovered, it is being processed to bronze table.

So a LIST request is needed to check the files in the source directory, and then this list of files is compared against the list of already processed files. The unprocessed files are then loaded into bronze table. For every file you need a GET request to read the file and its content.

Hey @filipniziol,

Thanks for the response, I've been trying to find some docs about this with no luck so far, do you happen to know any? 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now