topic Re: Error at model serving for quantised models using bitsandbytes library in Get Started Discussions

Error at model serving for quantised models using bitsandbytes library

phi_alpaca — Thu, 15 Feb 2024 16:13:01 GMT

Hello,

I've been trying to serve registered MLflow models at GPU Model Serving Endpoint, which works except for the models using bitsandbytes library. The library is used to quantise the LLM models into 4-bit/ 8-bit (e.g. Mistral-7B), however, it runs into error while registering at endpoint. This error is shown in the service log:

All libraries needed are registered in the requirements.txt files, it looks like one option to fix the error is to run a bash script to help it locate the right path of the package, but we're not able to do so at serving endpoint.

Has anyone successfully served a quantised LLM model at Databricks model serving using bitsandbytes? If so, how do you get around it? Any help on the topic would be much appreciated.

Thanks

Re: Error at model serving for quantised models using bitsandbytes library

G-M — Tue, 20 Feb 2024 09:02:18 GMT

Hi @phi_alpaca , we are facing exactly the same issue trying to serve a bitsandbytes quantized version of Mixtral-8x7B . Did you have any progress resolving this? The answer from @Retired_mod isn't too helpful and seems to be AI-generated...

As you say, the deployed container is such a black box that we can't take the diagnostic steps listed in the error output.

Re: Error at model serving for quantised models using bitsandbytes library

phi_alpaca — Tue, 20 Feb 2024 14:24:15 GMT

Hey @G-M , thanks for sharing your experience as well. Unfortunately I haven't had any luck on my end for resolving this. Would be interested to know if you have any breakthrough down the line. Is it something Databricks might be able to put a small fix in please? @Retired_mod
Thanks

Re: Error at model serving for quantised models using bitsandbytes library

JAgreenskylake — Thu, 22 Feb 2024 12:26:11 GMT

Hi, @phi_alpaca have you managed to solve this? We have a similar issue..

Re: Error at model serving for quantised models using bitsandbytes library

phi_alpaca — Thu, 22 Feb 2024 13:23:15 GMT

Hey @JAgreenskylake , no luck so far. I have been working around it by not using quantised models, which is not ideal, so really hope it's possible to do that soon.

Re: Error at model serving for quantised models using bitsandbytes library

G-M — Mon, 26 Feb 2024 18:13:13 GMT

@phi_alpaca

We have solved it by providing a conda_env.yaml when we log the model, all we needed was to add cudatoolkit=11.8 to the dependencies.

Re: Error at model serving for quantised models using bitsandbytes library

phi_alpaca — Tue, 27 Feb 2024 16:36:34 GMT

Thanks so much for sharing and glad it worked out for you guys!
I will have a go and feed back.

Re: Error at model serving for quantised models using bitsandbytes library

phi_alpaca — Thu, 07 Mar 2024 08:51:20 GMT

I seem to have some compatibility issues with cudatoolkit=11.8, would it be possible for you share what versions you use for torch, transformers, accelerate, and bitsandbytes? Thanks!

Re: Error at model serving for quantised models using bitsandbytes library

G-M — Thu, 07 Mar 2024 08:56:46 GMT

These versions are working for us:

torch==1.13.1
transformers==4.35.2
accelerate==0.25.0
bitsandbytes==0.41.3