Locally Hosted RAG (Retrieval-Augmented Generation)

What is the most valuable asset of a company? It's DATA.

The world’s most valuable data isn’t in the cloud, it’s locked behind firewalls, inside private servers. Retrieval-Augmented Generation (RAG) can be deployed on premises, close to the data, inside the security perimeter, and at enterprise scale.

Retrieval-Augmented Generation (RAG) significantly enhances the capabilities of language models by establishing dynamic connections to external knowledge repositories. Rather than operating exclusively within the constraints of pre-training data, RAG architectures systematically retrieve pertinent, current information from authoritative sources before generating responses. This methodology yields outputs characterized by superior accuracy, enhanced contextual relevance, and increased reliability.

According to industry projections for 2025, majority of enterprise artificial intelligence implementations are anticipated to integrate RAG or comparable knowledge grounding methodologies. This trend reflects a strategic organizational pivot toward deploying trustworthy, evidence-based AI systems that prioritize factual accuracy and source attribution in their outputs.

By implementing locally hosted RAG organizations can enhance their security, improve data protection, and reduce the risk of data breaches and other security threats. Here are some benefits:

1. Data Sovereignty:By hosting RAG locally, the business maintains complete control over its data, ensuring that sensitive information is not transmitted or stored outside the organization's premises.

2. Reduced Risk of Data Breaches:Locally hosting RAG minimizes the risk of data breaches associated with cloud-based services, as data is not transmitted over the internet and is less vulnerable to interception or hacking.

3. No Dependence on Third-Party Services:By hosting RAG locally, the business is not dependent on third-party services, which can be vulnerable to security risks, outages, or changes in terms of service.

4. Simplified Auditing and Logging:Locally hosting RAG makes it easier to implement auditing and logging mechanisms, ensuring that all interactions with the system are monitored and recorded.

Here is an illustration about this RAG implementation

Key Components

•Open-source models: Llama 3.2-3B

•Model formats: Llama-3.2-3B-Instruct-IQ3_M.gguf

•Search and Vectors: FAISS

•Text extraction: PyPDF2

•LLM framework: LANG Chain

Below is a complete Python implementation for a locally hosted Retrieval-Augmented Generation (RAG) application that processes PDF documents from a directory and uses Llama 3.2 3B as the language model.

1.Setting Up the Environment

First, let's set up the necessary dependencies. These packages provide the foundation for our RAG application, including document processing, vector storage, and model integration.

import os

# Set environment variables for offline mode

os.environ["HF_HUB_OFFLINE"] = "1" # Tell HuggingFace to work in offline mode

os.environ["TRANSFORMERS_OFFLINE"] = "1" # For transformers library

2. Importing Libraries

This imports all necessary components for document loading, text processing, vector storage, and model integration.

from langchain.text_splitter import RecursiveCharacterTextSplitter

from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader

from langchain_community.vectorstores import FAISS

from langchain_community.embeddings import HuggingFaceEmbeddings

from langchain.chains import RetrievalQA

from langchain_community.llms import LlamaCpp

import time

from tqdm import tqdm

import os.path

3. Loading PDF Documents

This function loads all PDF documents from the specified directory using PyPDFLoader, which is specifically designed for PDF documents.

def load_documents(directory_path="path://docs"):

# Replace with actual path if needed

directory_path = directory_path.replace("path://", "")

# Check if directory exists

if not os.path.exists(directory_path):

raise ValueError(f"Directory {directory_path} does not exist")

print(f"Loading documents from {directory_path}")

loader = DirectoryLoader(

directory_path,

glob="**/*.pdf",

loader_cls=PyPDFLoader,

show_progress=True

)

documents = loader.load()

print(f"Loaded {len(documents)} document pages")

return documents

4. Chunking Documents

Splitting documents into manageable chunks is crucial for effective retrieval. The RecursiveCharacterTextSplitter is used to create these chunks with appropriate overlap.

def split_documents(documents, chunk_size=1000, chunk_overlap=200):

text_splitter = RecursiveCharacterTextSplitter(

chunk_size=chunk_size,

chunk_overlap=chunk_overlap,

length_function=len

)

chunks = text_splitter.split_documents(documents)

print(f"Split into {len(chunks)} chunks")

return chunks

5. Creating Vector Embeddings

This function creates vector embeddings for document chunks using a Sentence Transformers model and stores them in a FAISS index for efficient similarity search.

def create_vector_store(chunks, embeddings_model_path="C:/AI/models/sentence-transformers/all-MiniLM-L6-v2"):

print(f"Creating embeddings using local model at {embeddings_model_path}")

# Initialize the embeddings model using local path

embeddings = HuggingFaceEmbeddings(

model_name=embeddings_model_path,

model_kwargs={"trust_remote_code": False} # Prevents checking for remote code

)

# Create vector store

print("Building FAISS index...")

vector_store = FAISS.from_documents(chunks, embeddings)

print("FAISS index built successfully")

return vector_store, embeddings

6. Setting Up the Llama 3.2 3B

This function initializes the Llama model using LlamaCpp, which provides efficient inference capabilities for large language models.

print("setting up Llama 3.2 3B model")

def setup_llm(model_path=None):

"""

Initialize the Llama 3.2 3B model.

You'll need to provide the path to the GGUF model file.

"""

if model_path is None:

raise ValueError("Model path must be provided for Llama 3.2 3b")

print(f"Loading Llama 3.2 3b model from {model_path}")

# Initialize the model with appropriate parameters

llm = LlamaCpp(

model_path=model_path,

temperature=0.1,

max_tokens=2048,

top_p=0.95,

n_ctx=4096, # Context window

verbose=False,

n_gpu_layers=-1, # Use all available GPU layers

n_batch=512, # Batch size for prompt processing

)

return llm

7. Building the RAG Pipeline

The RAG pipeline connects the vector store retriever with the language model, creating a system that can retrieve relevant document chunks and use them to generate accurate responses.

def build_rag_pipeline(vector_store, llm):

print("Building RAG pipeline...")

# Create the retrieval QA chain

qa_chain = RetrievalQA.from_chain_type(

llm=llm,

chain_type="stuff", # Other options: "map_reduce", "refine"

retriever=vector_store.as_retriever(

search_type="similarity",

search_kwargs={"k": 5} # Return top 5 most relevant chunks

),

return_source_documents=True,

verbose=True

)

return qa_chain

8. Complete Implementation

This function creates the complete RAG application by integrating all components and returns a query function that can be used to interact with the system.

def create_rag_application(docs_directory="C:/AI/", model_path=None,

embeddings_model_path="C:/AI/models/sentence-transformers/all-MiniLM-L6-v2",

build_option=None):

"""

Create a complete RAG application using Llama 3.2 3B and PDF documents.

Args:

docs_directory: Directory containing PDF documents

model_path: Path to the Llama 3.2 3B model file

embeddings_model_path: Path to local sentence transformers model

build_option: 1 for building from scratch, 2 for adding to existing index

Returns:

A function that can be used to query the RAG system

"""

# If build_option is not provided, ask the user

if build_option is None:

print("\nPlease select an option:")

print("1. Build FAISS index from scratch")

print("2. Add documents to existing FAISS index (if it exists)")

print("3. Use FAISS index as it is (no updates)")

while True:

try:

build_option = int(input("\nEnter your choice (1,2 or 3): "))

if build_option in [1, 2, 3]:

break

else:

print("Invalid option. Please enter 1 or 2 or 33.")

except ValueError:

print("Please enter a number.")

# Option 1: Build from scratch

if build_option == 1:

print("\nBuilding FAISS index from scratch...")

# Load and process documents

documents = load_documents(docs_directory)

chunks = split_documents(documents)

# Create vector store with local embeddings model

vector_store, embeddings = create_vector_store(chunks, embeddings_model_path)

# Option 2: Add to existing index

elif build_option == 2:

print("\nAttempting to add documents to existing FAISS index...")

# Try to load existing vector store

vector_store, embeddings = load_existing_vector_store(embeddings_model_path)

if vector_store is None:

print("No existing index found. Creating new index instead...")

documents = load_documents(docs_directory)

chunks = split_documents(documents)

vector_store, embeddings = create_vector_store(chunks, embeddings_model_path)

else:

# Add new documents to existing vector store

vector_store = add_documents_to_vector_store(vector_store, embeddings, docs_directory)

elif build_option == 3:

print("\nAttempting to load the existing FAISS index...")

# Try to load existing vector store

vector_store, embeddings = load_existing_vector_store(embeddings_model_path)

# Setup LLM

llm = setup_llm(model_path)

# Build RAG pipeline

qa_chain = build_rag_pipeline(vector_store, llm)

# Save vector store for future use

print("Saving FAISS index...")

vector_store.save_local("faiss_index")

print("faiss_index")

print("FAISS index saved successfully")

# Return a function that can be used to query the system2

def query_rag(question):

print(f"Query: {question}")

start_time = time.time()

result = qa_chain({"query": question})

end_time = time.time()

print(f"Time taken: {end_time - start_time:.2f} seconds")

# Return both the answer and source documents

return {

"answer": result["result"],

"source_documents": result["source_documents"]

}

return query_rag

9. Example Usage

This example demonstrates how to use the RAG application with test queries to verify its accuracy and functionality.

if __name__ == "__main__":

# Path to the Llama 3.2 3B model (GGUF format)

MODEL_PATH = "C:/AI/llama3_2/Llama-3.2-3B-Instruct-IQ3_M.gguf"

# Path to local embedding model (you need to download this in advance)

EMBEDDINGS_MODEL_PATH = "C:/AI/models/sentence-transformers/all-MiniLM-L6-v2"

# Check if embeddings model exists

if not os.path.exists(EMBEDDINGS_MODEL_PATH):

print(f"WARNING: Embeddings model not found at {EMBEDDINGS_MODEL_PATH}")

print("You need to download the model first. Run this command while online:")

print(

"python -c \"from huggingface_hub import snapshot_download; snapshot_download('sentence-transformers/all-MiniLM-L6-v2', local_dir='C:/AI/models/sentence-transformers/all-MiniLM-L6-v2')\"")

exit(1)

# Create the RAG application with user choice for building index

rag_query = create_rag_application(

docs_directory="C:/AI/",

model_path=MODEL_PATH,

embeddings_model_path=EMBEDDINGS_MODEL_PATH,

build_option=None # This will prompt the user to choose

)

# Example queries to test the system

test_questions = [

"what is unsupervised learning?",

"What are The Four Stages of the Design Process?",

"What is a Multi - Layer Perceptron?"

]

# Ask if user wants to run test queries

run_tests = input("\nDo you want to run test queries? (y/n): ").lower().strip()

if run_tests == 'y':

# Run test queries

for question in test_questions:

print("\n" + "=" * 50)

result = rag_query(question)

print("\nQuestion:", question)

print("\nAnswer:", result["answer"])

print("\nSources:")

for i, doc in enumerate(result["source_documents"][:3]): # Show top 3 sources

print(f"Source {i + 1}:", doc.metadata.get("source", "Unknown"),

f"(page {doc.metadata.get('page', 'Unknown')})")

print("=" * 50 + "\n")

# Interactive mode

print("\nEntering interactive mode. Type 'exit' to quit.")

while True:

user_question = input("\nEnter your question: ")

if user_question.lower() == 'exit':

break

print("\n" + "=" * 50)

result = rag_query(user_question)

print("\nQuestion:", user_question)

print("\nAnswer:", result["answer"])

print("\nSources:")

for i, doc in enumerate(result["source_documents"][:3]): # Show top 3 sources

print(f"Source {i + 1}:", doc.metadata.get("source", "Unknown"),

f"(page {doc.metadata.get('page', 'Unknown')})")

print("=" * 50 + "\n")

Important Considerations

The full llama 3.2 405B parameter model is extremely large, even in quantized GGUF format, requiring significant storage space and powerful hardware to run this implementation uses Llama 3.2-3B to make things easier in a laptop without GPU's
Most repositories offer various quantization levels (like Q8_0, Q6_K, Q5_K, Q4_K) which trade off quality for reduced size and memory requirements
For most consumer hardware, you'll want to use a more heavily quantized version unless you have enterprise-grade GPUs with sufficient VRAM
Always verify you're downloading from a reputable source to avoid security risks
Be aware of the license terms for using Llama 3.2 models, even in GGUF format

Here are some applications for locally hosted RAG systems:

Enterprise and Business Applications

Customer Service Automation: Companies are implementing locally hosted RAG systems to provide instant, accurate customer support by retrieving information from their internal knowledge bases and documentation

Sales Assistance and Lead Generation: RAG systems help sales teams by providing real-time access to product information, pricing data, and customer history stored locally within the organization

Industry-Specific Applications

Legal Research and Document Analysis: Law firms are deploying local RAG systems to enhance legal research capabilities, allowing lawyers to query case law, statutes, and internal documents while maintaining client confidentiality. These systems democratize AI technology for smaller law firms and independent practitioners

Healthcare and Medical Applications: In specialized medical fields, locally hosted RAG tools create assistants that combine LLM capabilities with authoritative domain-specific medical literature and patient data, ensuring HIPAA compliance

Automotive Industry: Research shows effective optimization schemes for local RAG deployment using Ollama in automotive industrial applications, particularly for processing technical documentation and maintenance procedures