cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

auto loader

JissMathew
Contributor III
source_df = (spark.readStream
                 .format("cloudFiles")
                 .option("cloudFiles.format", "csv")
                 .option("header", "true")
                 .option("timestampFormat", "d-M-y H.m")
                 .option("cloudFiles.schemaLocation", f"{landing_folder_path}/Opportunity_schema")
                 #.option("cloudFiles.inferColumnTypes", "true")
                 .schema(schema)  # Explicitly define schema to avoid invalid characters
                 .load(f"{landing_folder_path}/Opportunity")
    )
    source_df = source_df.filter("Id IS NOT NULL")  # Example of filtering out corrupt data
    write_query = (source_df.writeStream
                   .format("delta")
                   .option("checkpointLocation", f"{landing_folder_path}/Opportunity/checkpoint")
                   .option("mergeSchema", "false")
                   .outputMode("append")
                   .trigger(availableNow=True)
                   .toTable("dev.demo_db.Opportunity_raw")
    )

    write_query.awaitTermination()  # Ensure the stream is running

ingest()
i have issue with auto loader that i can't getting incremental load on this . if rerun the ingest() the some unstructured data ingecting into  Opportunity_raw
Jiss Mathew
India .
2 REPLIES 2

cgrant
Databricks Employee
Databricks Employee

Hi JissMatew,

As long as your checkpointLocation is not deleted and does not change, you should receive an incremental feed when loading data. Please verify that the checkpoint is not being deleted or moved between runs. The checkpoint is how the stream keeps track of its progress.

If you're still noticing this, please give more details about the duplicate unstructured data that you are ingesting.

NandiniN
Databricks Employee
Databricks Employee

In case of failures, Auto Loader can resume from where it left off by information stored in the checkpoint location and continue to provide exactly-once guarantees when writing data into Delta Lake. You don’t need to maintain or manage any state yourself to achieve fault tolerance or exactly-once semantics.

 

https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/index.html#how-does-auto-l...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group