topic Re: Autoloader ingnore one folder in path in Data Engineering

Autoloader ignore one folder in path

deng_dev — Tue, 21 May 2024 14:39:14 GMT

Hi everyone!

I am trying to setup Autoloader to read json file with specific name from all subfolders under the path except one.

Could someone advice how this can be achieved?

For example, I need to read from .../*/specific_name.json, but ignore test folder

So ../test/../specific_name.json should be ignored

I have tried this GLOB expression, but seems that this one is not working:

"<bucket_name>/!(*test*)/specific_name.json"

Re: Autoloader ingnore one folder in path

standup1 — Tue, 21 May 2024 14:01:12 GMT

I think you can use REGEXP to achieve this. This might not be the best way, but it should get the job done. It's all about filtering that file in the df from getting loaded. Try something like this
df.select(“*”,”_metadata”).select(“*”,”_metadata.file_path”).filter(“file_path not REGEXP ‘/folder_name/’ “)

Make sure you change the folder name and df to yours.

You can also look into (pathGlobfilter) if the file name is static.

Re: Autoloader ingnore one folder in path

deng_dev — Tue, 21 May 2024 14:41:44 GMT

Hi @standup1 , thank you for the response. The problem is I need to filter those files before they are loaded with Autoloader, as I need Autoloader to ignore them in schema inference.

Re: Autoloader ingnore one folder in path

standup1 — Tue, 21 May 2024 15:01:22 GMT

oh I see, I know this is one of the limitation of the Autoloader. However, if your file path is different from test and static. you can increase your path in load() to make sure the load doesn't read from test. for example.

If your prod path is ...prod/*/specific_name.json and your test path is ...test/*/specific_name.json. Then, in load() do load(" ...prod/*/specific_name.json"). Whatever distinguishes your other path from test is what you use in load() so it will skip the test one. I hope this help or if you can share your full path of the files that would be helpful to see.