topic Re: Unzipping with Serverless Compute in Data Engineering

Unzipping with Serverless Compute

Nathant93 — Fri, 14 Jun 2024 14:56:16 GMT

I have started using serverless compute but have come across the limitation that I cannot use the local filesystem for temporarily storing the files and directories before moving them to where they need to be in adls.

Does anyone have a way of unzipping folders using serverless compute?

Thanks

Re: Unzipping with Serverless Compute

delonb2 — Sat, 15 Jun 2024 04:33:13 GMT

Do you have the ability to make a Unity Catalog volume? You could use it as temporary storage before migrating the files to adls.

Re: Unzipping with Serverless Compute

Nathant93 — Mon, 17 Jun 2024 10:14:37 GMT

I do have that ability and it has worked thanks for the suggestion - I had thought volumes had to be linked to a storage container, didn't realise that they could just work as a temporary location.