Download Dolly model on local machine

[FutureLegend]

Hi~ I am new to LLM engineering, and am trying to download the Dolly-v2-7b model on local machine, so I don't need to connect to internet each time I am going to run the Dolly-v2-7b. Is it possible to do that? Thanks a lot!

[Kaniz]

Hi @FutureLegend, To download the Dolly-v2-7b model on your local machine, you can use MLflow Export-Import to migrate MLflow models from one workspace to another. You can copy the model from one workspace to another, from a development to a production workspace.

However, it would be best if you had MLflow 2.3 installed, and any cluster with the Hugging Face transformers library installed can be used for batch inference. The transformers library comes preinstalled on 10.4 LTS ML and above.

Here are the steps you can follow:

1. Export the Dolly-v2-7b model from your Databricks workspace using MLflow Export-Import.
2. Download the exported model to your local machine.
3. Install the Hugging Face transformers library on your local machine.
4. Load the exported model to your local machine using the transformers library.

Sources:
- https://docs.databricks.com/machine-learning/manage-model-lifecycle/workspace-model-registry.html
- https://docs.databricks.com/machine-learning/train-model/huggingface/model-inference-nlp.html

[sean_owen]

@Kaniz this is not necessary for downloading a model; it actually has nothing to do with Databricks.

Dolly is hosted on Hugging Face. If you download it via transformers, it will actually cache it and not download again, already. But you can always just download the files of any model to a local dir: https://huggingface.co/databricks/dolly-v2-12b/tree/main You then load that local dir as a model.

[FutureLegend · Accepted solution]

Hi Kaniz and Sean, thanks for your responses and time.
I was trying Kaniz's method, but got a reply from Sean, so I tried that too. I downloaded the file from the link Sean provided and saved it on my local machine, then used the code for Dollyv2 (https://huggingface.co/databricks/dolly-v2-3b) and ran it successfully without connecting to Huggingface/ Databricks. Haven't tried without internet, but I think it will work too. Thanks again Kaniz and Sean for your replies!

code I used:

pip install "accelerate>=0.16.0,<1" "transformers[torch]>=4.28.1,<5" "torch>=1.13.1,<2"

import torch

from transformers import pipeline

generate_text = pipeline(model="local_machine/path/to/downloaded_Dolly_folder", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")

res = generate_text("Explain to me the difference between nuclear fission and fusion.")

print(res[0]["generated_text"])