topic Re: MLlib load from UC Volume: IllegalArgumentException: Cannot access the UC Volume path... in Data Governance

MLlib load from UC Volume: IllegalArgumentException: Cannot access the UC Volume path...

stiaangerber — Wed, 22 Nov 2023 06:41:59 GMT

I'm trying to store MLlib instances in Unity Catalog Volumes. I think volumes are a great way to keep things organized.

I can save to a volume without any issues and I can access the data using spark.read and with plain python open(). However, when I try to load a saved MLlib instance using MLReader.load, I get:

IllegalArgumentException: Cannot access the UC Volume path from this location.

See the attached demo. I'm a metastore admin and owner of the volume.

Am I missing something? Is this not supported (yet)? Can I make it work somehow?

I've tested this on DBR 13.3 and 14.2, same thing. Any help will be much appreciated.

Re: MLlib load from UC Volume: IllegalArgumentException: Cannot access the UC Volume path...

stiaangerber — Wed, 22 Nov 2023 07:31:18 GMT

Thanks @Retired_mod. For now, I implemented a workaround. For others with a similar issue, this works:

def load_model(mlclass, uc_vol_path): try: tmpdir = f'/FileStore/{uuid.uuid4().hex}' dbutils.fs.cp(uc_vol_path, tmpdir, recurse=True) model = mlclass.load(tmpdir) return model finally: dbutils.fs.rm(tmpdir, recurse=True) dct2 = load_model(DCT, '/Volumes/demo/default/vol/dct')

Re: MLlib load from UC Volume: IllegalArgumentException: Cannot access the UC Volume path...

stiaangerber — Wed, 22 Nov 2023 12:13:13 GMT

Actually, the above is no good for models with associated data. (You can only delete the tmpdir after your done using the model)

Re: MLlib load from UC Volume: IllegalArgumentException: Cannot access the UC Volume path...

slimexy — Thu, 23 Nov 2023 07:01:07 GMT

Just to supplement that if the ML model is saved and then loaded within the same execution, calling load() will not cause the mentioned exception. Copying the model directory from UC volume to ephemeral storage attached to the driver node is also a work around (without the need to delete the tmpdir in DBFS after loading the model), but works in single node mode only.