Hello @K_Anudeep ,

As I mentioned, we realized that the special character encoding and decoding was broken. So we had solved the issue as the following by changing the path string in autoloader:

old path: s3://my-sink/prod/app_daily/*/*/*/*.json.gz

After changing the path string, a clear run with the option ignoreMissingFiles=true cleared the state. After, we removed the option and pipeline continued to run successfully.

Regardless, thanks for your clear answer.

PS: I do not know our path string was the best regarding the conventions but it was working anyway. If you can suggest better and performant string formats, I appreciate any help. Thanks.