cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

how to use databricks foundation models outside databricks environment?

Onkar01
New Contributor

i want to use below code outside databricks environment without using cluster .getting issue while i am using chat_model.predict().

how to authenticate this outside databricks or any other way to this? 

 

# Test Databricks Foundation LLM model
from langchain.chat_models import ChatDatabricks
chat_model = ChatDatabricks(endpoint="databricks-llama-2-70b-chat", max_tokens = 200)
print(f"Test chat model: {chat_model.predict('What is Apache Spark')}")
 
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
from langchain.chat_models import ChatDatabricks

TEMPLATE = """You are an assistant for Databricks users. You are answering python, coding, SQL, data engineering, spark, data science, DW and platform, API or infrastructure administration question related to Databricks. If the question is not related to one of these topics, kindly decline to answer. If you don't know the answer, just say that you don't know, don't try to make up an answer. Keep the answer as concise as possible.
Use the following pieces of context to answer the question at the end:
{context}
Question: {question}
Answer:
"""
prompt = PromptTemplate(template=TEMPLATE, input_variables=["context", "question"])

chain = RetrievalQA.from_chain_type(
    llm=chat_model,
    chain_type="stuff",
    retriever=get_retriever(),
    chain_type_kwargs={"prompt": prompt}
)
llm_output = chain(kwargs, return_only_outputs=True)
1 REPLY 1

Wojciech_BUK
Valued Contributor III

Are you perhaps talking about Model Serving ?

https://docs.databricks.com/en/machine-learning/model-serving/index.html#requirements

This is how you configure it 

https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html

This is how you use it via REST API

https://docs.databricks.com/en/machine-learning/model-serving/score-model-serving-endpoints.html#cha...

NOTE - you will use cluster - Serverless for model serving - that will allow you to make online scoring 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group