Autoloader ignore one folder in path

[deng_dev]

Hi everyone!

I am trying to setup Autoloader to read json file with specific name from all subfolders under the path except one.

Could someone advice how this can be achieved? 

For example, I need to read from .../*/specific_name.json, but ignore test folder

So ../test/../specific_name.json should be ignored

I have tried this GLOB expression, but seems that this one is not working: 

"<bucket_name>/!(*test*)/specific_name.json"

[standup1]

I think you can use REGEXP to achieve this. This might not be the best way, but it should get the job done. It's all about filtering that file in the df from getting loaded. Try something like this
df.select(“*”,”_metadata”).select(“*”,”_metadata.file_path”).filter(“file_path not REGEXP ‘/folder_name/’ “)

Make sure you change the folder name and df to yours.

You can also look into (pathGlobfilter) if the file name is static.

[deng_dev]

Hi @standup1 , thank you for the response. The problem is I need to filter those files before they are loaded with Autoloader, as I need Autoloader to ignore them in schema inference.

[standup1]

oh I see, I know this is one of the limitation of the Autoloader. However, if your file path is different from test and static. you can increase your path in load() to make sure the load doesn't read from test. for example.

If your prod path is ...prod/*/specific_name.json and your test path is ...test/*/specific_name.json. Then, in load() do load(" ...prod/*/specific_name.json"). Whatever distinguishes your other path from test is what you use in load() so it will skip the test one. I hope this help or if you can share your full path of the files that would be helpful to see.