File trigger options -- cloudFiles.allowOverwrites

srinivas_001 — Fri, 15 Mar 2024 15:27:21 GMT

I have a Job configured to run on the file arrival
I have provided the path as
File arrival path: s3://test_bucket/test_cat/test_schema/

When a new parquet file arrived in this path the job was triggering automatically and processed the file

In case of reloading means, Overwriting the existing file
I am uploading the same file again (with same name) to this path then No run was triggered
(No worries about duplicating the data I just need to trigger the job)
Code as below:
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "parquet")
.option("inferSchema", "false")
.option("cloudFiles.allowOverwrites", "true")
.option("cloudFiles.schemaLocation", "checkpoint_dir")
.load(data_source)

Do I need to enable any other settings in order to trigger the job?

Re: File trigger options -- cloudFiles.allowOverwrites

srinivas_001 — Mon, 18 Mar 2024 10:40:55 GMT

Hi Kaniz,

Thank you for the response.
I am using the databricks runtime 11.3, also checked the checkpoint and data source location which are properly configured. Still I am unable to trigger the job.

NOTE: Incoming files are pushed to AWS s3 location from Apache airflow with REPLACE option TRUE.

topic File trigger options -- cloudFiles.allowOverwrites in Data Engineering

File trigger options -- cloudFiles.allowOverwrites

Re: File trigger options -- cloudFiles.allowOverwrites