Hi @Kaniz , Thanks your response. I did a couple of your recommendations and no look so far. What I did so far:
Check Model Configuration:
- Ensure that youโve correctly configured the model. Double-check the settings related to scale_to_zero_enabled, workload_type, and workload_size. Make sure they match your intended setup.
R: I did the configuration similar ,not equal, comparing with 02-Deploy-RAG-Chatbot-Model (LLM with rag on databricks - dbdemos). lets show to you what I did on this subject:
w = WorkspaceClient()
endpoint_config = EndpointCoreConfigInput(
name=serving_endpoint_name,
served_models=[
ServedModelInput(
model_name=model_name,
model_version=latest_model_version,
workload_size="Small",
workload_type="GPU_SMALL",
scale_to_zero_enabled=False,
environment_vars={
"DATABRICKS_TOKEN": "{{secrets/kb-kv-secrets/adb-kb-ml-token}}", # <scope>/<secret> that contains an access token
}
)
]
)
Also Iยดm using FAISS as vector search with FAIS GPU package.
model_name = "sentence-transformers/all-mpnet-base-v2"
model_kwargs = {"device": "cuda:0"}
embeddings = HuggingFaceEmbeddings(model_name=model_name, model_kwargs=model_kwargs)
# storing embeddings in the vector store
vectorstore = FAISS.from_documents(all_splits, embeddings)
and with save_local and load
#Persist to be ready for mlflow
persist_directory = "langchain/faiss_index"
vectorstore.save_local(persist_directory)
def get_retriever(persist_dir: str = None):
if (persist_dir==None):
db = FAISS.load_local("langchain/faiss_index", embeddings)
else:
db = FAISS.load_local(persist_dir, embeddings)
return db.as_retriever()
Model Name Mapping:
- Sometimes, errors like this occur because the model name isnโt included in the model_token_mapping dictionary. To resolve this, add your model (e.g., โgpt-35-turbo-16kโ) to the dictionary along with its correspo....
R: I donยดt even know how to implement this. Seams to be a static function, but where I put the code bellow. Do you have a tip ?
@staticmethod
def modelname_to_contextsize(modelname: str) -> int:
model_token_mapping = {
# ... existing model mappings ...
"gpt-35-turbo-16k": <max_context_size_for_this_model>, # Add your model here
}
# rest of the method..Output Format Alignment:
- Verify that the output format of your LLM aligns with what your agent expects. If necessary, adjust the parsing logic to handle the specific output format of your custom LLM.
R: I think that is Ok, I put on this way and when I test in the Databricks notebook it works fine.
def transform_output(response):
return str(response)
llm3 = Databricks(endpoint_name='mm-v2-llama2-7b-chat-hf',extra_params={"temperature":0.0001,"max_tokens": 120},transform_output_fn=transform_output) #SAME RESULT
#llm = Databricks(endpoint_name='mm-v2-llama2-7b-chat-hf',extra_params={"temperature":0.0001,"max_tokens": 120})
#input_text = "What is apache spark?"
input_text = "Qual o tipo do campo WarehouseBalance ?"
print(llm3.predict(input_text))
Prompt Assignment:
- When iterating over LLM models, try assigning the prompt inline instead of using a variable. For example:chain = LLMChain(llm=llm_model, prompt=PromptTemplate(template=template, input_variables=['context', 'prompt']))
R: I think that is OK, I put on this way:
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
TEMPLATE = """You are an assistant for Databricks users. You are answering python, coding, SQL, data engineering, spark, data science, DW and platform, API or infrastructure administration question related to Databricks. If the question is not related to one of these topics, kindly decline to answer. If you don't know the answer, just say that you don't know, don't try to make up an answer. Keep the answer as concise as possible.
Use the following pieces of context to answer the question at the end:
{context}
Question: {question}
Answer:
"""
prompt = PromptTemplate(template=TEMPLATE, input_variables=["context", "question"])
chain = RetrievalQA.from_chain_type(
llm=llm3,
chain_type="stuff",
retriever=get_retriever(),
chain_type_kwargs={"prompt": prompt}
)
Some actions Iยดm planning to do:
1) Implement another vector search such as chroma (I think it will not use GPU)
2) Implement the Model Name mapping. However I donยดt know where I put the code.
Any thoughts ?
Thanks
