According to the docs you linked, the glob filter on input-path only works on directories, not on the files themselves.

So if you want to filter on certain files in the concerning dirs, you can include an additional filter through the pathGlobFilter option:

.option("pathGlobFilter", "*_INPUT")

https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-s3.html#use-cloudfiles-sou...