how to use databricks foundation models outside databricks environment?

[Onkar01]

i want to use below code outside databricks environment without using cluster .getting issue while i am using chat_model.predict().

how to authenticate this outside databricks or any other way to this? 

# Test Databricks Foundation LLM model

from langchain.chat_models import ChatDatabricks

chat_model = ChatDatabricks(endpoint="databricks-llama-2-70b-chat", max_tokens = 200)

print(f"Test chat model: {chat_model.predict('What is Apache Spark')}")

from langchain.chains import RetrievalQA

from langchain.prompts import PromptTemplate

from langchain.chat_models import ChatDatabricks

TEMPLATE = """You are an assistant for Databricks users. You are answering python, coding, SQL, data engineering, spark, data science, DW and platform, API or infrastructure administration question related to Databricks. If the question is not related to one of these topics, kindly decline to answer. If you don't know the answer, just say that you don't know, don't try to make up an answer. Keep the answer as concise as possible.

Use the following pieces of context to answer the question at the end:

{context}

Question: {question}

Answer:

"""

prompt = PromptTemplate(template=TEMPLATE, input_variables=["context", "question"])

chain = RetrievalQA.from_chain_type(

    llm=chat_model,

    chain_type="stuff",

    retriever=get_retriever(),

    chain_type_kwargs={"prompt": prompt}

)

llm_output = chain(kwargs, return_only_outputs=True)

[Wojciech_BUK]

Are you perhaps talking about Model Serving ?

https://docs.databricks.com/en/machine-learning/model-serving/index.html#requirements

This is how you configure it 

https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html

This is how you use it via REST API

https://docs.databricks.com/en/machine-learning/model-serving/score-model-serving-endpoints.html#cha...

NOTE - you will use cluster - Serverless for model serving - that will allow you to make online scoring