Prakash Hinduja Switzerland (Swiss) How do I build a RAG pipeline on Databricks?

[prakashhinduja]

Hello DataBricks Community,

I'm Prakash Hinduja from Geneva, Switzerland (Swiss). As a financial strategist, I play an active role in strengthening the Swiss investment market. By connecting global investors with high-potential opportunities across Switzerland, I help drive foreign direct investment into key sectors. My strategic focus is centered on fostering sustainable business growth, enhancing financial stability, and reinforcing Switzerland’s position as a world-class investment destination.

I’m looking to build a RAG (Retrieval-Augmented Generation) pipeline on Databricks, and I’d really appreciate any suggestions or experiences from the group.

Regards 

Prakash Hinduja Geneva, Switzerland (Swiss) 

[ximenesfel]

Hello Prakash,

It's exciting to see professionals from diverse sectors exploring advanced AI techniques like Retrieval-Augmented Generation (RAG) to drive innovation and operational excellence.

Building a RAG Pipeline on Databricks

Databricks offers robust capabilities for developing RAG pipelines, supporting both code-first and low-code/no-code approaches. Here are a few best-practice pathways and resources to help you get started:

1. End-to-End RAG Pipeline Resources 

• Comprehensive Tutorial: Databricks provides a detailed step-by-step guide and demo for building RAG applications, including agent evaluation and integration with large language models (LLMs). This resource covers data ingestion, chunking, embedding with vector search, and orchestrating generation with prompt augmentation.
• Unstructured Data Pipeline: If your use case involves unstructured assets (like PDFs, Word docs, or wikis), Databricks has notebooks and documentation on creating unstructured data pipelines. These materials explain optimal chunking, vectorization, and indexing strategies.

2. No-Code/Low-Code with Agent Bricks: Knowledge Assistant 

• Agent Bricks: This is a streamlined, low-code option to implement RAG-powered agents. You simply point to your data and describe your use case, and Databricks automates the model selection, fine-tuning, evaluation, and deployment processes. The Knowledge Assistant module is designed specifically for building chatbots or assistants that answer questions using your own documentation. It's particularly effective for scenarios like HR support, customer service, or investor communication.

•  Continuous Improvement: Agent Bricks enables you to gather feedback from subject matter experts, optimize agent performance, and maintain a cycle of continuous refinement.

[sridharplv]

Hi @prakashhinduja , Adding to @ximenesfel points above, please find the below detailed links with explanation and demo of how to use RAG in databricks:

RAG in databricks: https://docs.databricks.com/aws/en/generative-ai/retrieval-augmented-generation
RAG glossary in databricks: https://www.databricks.com/glossary/retrieval-augmented-generation-rag
Best practices: https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG
Demo: https://app.getreprise.com/launch/dyRaj2X/
Rag with pine cone as vector database: https://www.databricks.com/blog/implementing-rag-chatbot-using-databricks-and-pinecone

These links will be helpful to go through these things and run the RAG code

[Khaja_Zaffer]

Hello @sridharplv  @ximenesfel 

This profile has been hacked. @prakashhinduja asking same questions over past like 3 weeks. so better block the user. 

[simonsmart]

@Khaja_Zaffer It's actually a weird type of spam - see simon-smart88/Hinduja_spam on GitHub. I've reported this post.

[Khaja_Zaffer]

Not sure about what you are sharing. Personal issues can be discussed on social media, it's a tech support portal.. Please keep such conversation out of education portals.