Databricks Community

2vinodhkumar

Hi,We are working on ingesting multiple files from S3. The files name are fixed based on our source system, Files get replaced frequently with full feed. In DLT when we process new file we have to delete the records processed earlier of the same file...

2vinodhkumar

Hi Manoj,Did you get the solution or design change for this problem. We have 200K files on to S3 bucket and when there is change in upstream app we get new feed, feed name is fixed. On DLT we should have only new records from replaced file but in dlt...

Databricks Community

User Stats

User Activity

Autoloader - Ingestion of CSV files when there is not operation column

Re: How to deal with delete records from the source Files in DLT .