Multi-Agent RAG · LLM Evaluation · FastAPI · Offline Llama3

AI Product Feedback Intelligence

FeedbackIntel ingests user reviews from App Store, Play Store, G2, NPS surveys, and support tickets into a FAISS vector index, then runs a three-agent RAG pipeline to answer product questions with structured, cited insight reports. The Planner decomposes each question into investigation themes, the Retriever fetches the most semantically relevant feedback per theme using nomic-embed-text embeddings, and the Synthesizer writes a grounded report backed by verbatim quotes with full source attribution. The entire stack runs offline via Ollama, with no API costs and no data leaving the server.

Try the Live Demo

3-agent pipeline: Planner, Retriever, Synthesizer Real Play Store and App Store reviews 3 custom LLM-as-judge eval metrics FAISS semantic search with metadata filters Continuous ingestion pipeline with PostgreSQL tracking $0 inference cost, fully offline

Python 3.11 FastAPI Llama3 (Ollama) nomic-embed-text FAISS PostgreSQL SQLAlchemy 2.0 asyncpg Docker GitHub Actions

Impact

Key Outcomes

Quantified results across agent pipeline performance, evaluation quality, and infrastructure design decisions that shaped the system.

Specialized agents in the orchestration pipeline

Max unique feedback items analyzed per insight query (4 themes × 12)

0.82

Average faithfulness score on Llama3 at 1,000+ items indexed

Inference cost per report � fully offline Ollama stack

Problem It Solves

Product teams receive feedback from 8+ channels with no unified view. Answering "why do users churn?" requires days of manual synthesis across App Store reviews, G2 posts, NPS verbatims, and support tickets. FeedbackIntel replaces that with a single API call that returns a cited, structured report in under 2 minutes.

Why Offline Llama3

Customer emails, NPS comments, and support tickets contain PII. Sending them to a cloud LLM API violates most enterprise data policies. Running Llama3 locally via Ollama provides GPT-class reasoning with zero data egress � the entire stack runs on a single server with no external dependencies.

Specificity: The Custom Metric

Standard RAGAS metrics (faithfulness + relevance) catch hallucinations and off-topic reports, but not vagueness. A report saying "users are unhappy with checkout" is faithful and relevant but completely useless. The custom specificity metric penalizes exactly this and rewards cited, attributed, quantified claims.

System Design

End-to-End Architecture

Two phases: an async ingestion pipeline that embeds feedback into FAISS and tracks state in PostgreSQL, and a query-time agent pipeline that converts a product question into a cited insight report with evaluation scores attached.

Shared state dict flows between agents. Each agent receives a Python dict and returns it updated. Planner adds themes, Retriever adds formatted_context and cited_ids, Synthesizer adds report. No message queue, no shared memory � just a plain dict passed through an async sequential pipeline. Easy to test, easy to extend with a 4th agent.

FAISS is reconstructible from PostgreSQL at any time. Every indexed item has embedded=True in the DB. On server restart, FAISS is empty but the pipeline re-runs automatically and re-embeds all items. This means FAISS is a pure cache layer � PostgreSQL is the source of truth. No separate FAISS persistence file to manage.

Live Demo

Checking...

Ask the RAG Pipeline

Ask any product question in plain English. The Planner decomposes it into investigation themes, the Retriever searches real Play Store and App Store reviews indexed in FAISS, and the Synthesizer writes a cited insight report scored by an LLM-as-judge evaluator. This runs live on my machine via a Cloudflare Tunnel.

Ask about product feedback

Try an example

What are the main checkout issues customers are reporting?

Why are customers churning in their first week?

What features are customers most requesting?

What are customers saying about the mobile experience?

Summarize the positive feedback and what customers love

What payment and billing problems are users facing?

What are the main onboarding friction points?

What are the top themes in 1-star reviews?

Retrieving relevant feedback from FAISS...

Identified Themes

Insight Report

Llama

LLM-as-Judge Evaluation

Auto-scored

Question

→

Planner

→

FAISS Retrieval

→

Synthesizer

→

LLM-as-Judge Eval

Retrieval-Augmented Generation

RAG Pipeline: From Raw Feedback to Cited Report

Every insight report is grounded entirely in retrieved user feedback. nomic-embed-text converts all indexed items into 768-dimensional vectors offline at ingestion time. At query time, the Planner's investigation themes are embedded with the same model and searched against the FAISS index. Only retrieved evidence enters the Synthesizer's context window � the LLM has no access to any knowledge beyond what was fetched from the index for that specific query.

Step 01

Ingest and Store

Feedback arrives via API with structured metadata: source platform (App Store, Play Store, G2, Capterra, NPS, support ticket, Twitter), star rating (1-5), category, date, and free-text content. Items are persisted to PostgreSQL with embedded=False, flagging them as pending vector indexing.

Step 02

Generate Embeddings

The ingestion pipeline fetches unembedded items in batches of 50 and calls nomic-embed-text via Ollama's local embedding API. Each item's text is converted to a 768-dimensional float vector, then L2-normalized so that inner product equals cosine similarity during retrieval, removing the need for any normalization step at query time.

Step 03

Index in FAISS

Normalized vectors are added to a FAISS IndexFlatIP index held in-memory. A parallel Python list at matching row indices stores each item's metadata: id, source, rating, and date. An asyncio.Lock serializes all writes to prevent concurrent mutations from multiple FastAPI workers. The embedded flag in PostgreSQL is set to True after each batch completes.

Step 04

Decompose into Themes

At query time, the Planner Agent prompts Llama3 to decompose the product question into up to 4 focused investigation themes. A broad question like "why do users churn?" becomes specific, searchable angles such as "checkout failure," "slow performance," "missing features," and "account issues." Each theme gets its own independent retrieval pass against the FAISS index.

Step 05

Retrieve per Theme

Each theme string is embedded using nomic-embed-text and searched against FAISS. The system over-fetches 4 times the requested k before applying source or rating filters, ensuring k results survive even when the corpus is skewed toward a particular source. The top-12 items per theme are returned after filtering, then deduplicated by ID across all themes � the same review is never cited twice.

Step 06

Assemble Context Window

All retrieved items are formatted with full attribution: [App Store, 2?, 2024-03-10] followed by the verbatim review text. This formatted block is the only input the Synthesizer receives alongside the original question. The prompt explicitly prohibits any claim not traceable to this context, making every finding in the final report auditable back to a specific user review.

nomic-embed-text

An open-source embedding model running locally via Ollama with zero data egress. Produces 768-dimensional vectors that match or exceed OpenAI text-embedding-ada-002 on semantic similarity benchmarks. Embeddings are generated at ingestion time, so query latency is not affected by embedding throughput � all vectors are already in FAISS before any question is asked.

FAISS IndexFlatIP

Exact nearest-neighbor search using inner product on L2-normalized vectors � no approximation error. Appropriate at product feedback scale (under 100K items). The index lives entirely in-memory and is fully reconstructible from PostgreSQL at startup. Metadata is kept in a parallel in-memory list indexed to match FAISS row numbers, enabling filtered retrieval without a secondary lookup.

Grounded Context Window

The Synthesizer receives only the retrieved feedback and the original question. No system knowledge, no conversation history, no web access. This hard constraint is what makes faithfulness measurable: any claim the model makes that cannot be traced to the retrieved context is catchable by the evaluator. The context window holds up to 48 attributed feedback items (4 themes � 12 results, deduplicated).

RAG over fine-tuning for feedback analysis. Fine-tuning would bake review knowledge into model weights, making it expensive to update as new feedback arrives. With RAG, new reviews are reflected in the next query the moment they are embedded and indexed � no retraining, no deployment cycle. The same Llama3 model serves all product corpora; only the FAISS index changes.

Per-theme retrieval beats single-query retrieval for multi-faceted questions. A single embedding of the full question biases FAISS toward whichever aspect has the strongest signal. Decomposing into themes and running a separate search per theme gives all angles equal representation in the context window. A question covering both positive and negative user signals gets independent retrieval passes for each direction.

Agent Design

Three-Agent Orchestration Pipeline

Each agent is a Python class extending BaseAgent with a single async run(state: dict) -> dict method. The orchestrator runs them sequentially, passing the shared state dict through. No framework dependency � the pattern is intentionally simple so each agent is independently testable.

■ Agent 1

Planner Agent

Receives the raw product question and uses Llama3 to decompose it into up to 4 specific, searchable investigation themes. The prompt enforces JSON array output. The decomposition step is critical: without it, a question like "why do users churn?" retrieves a mix of everything instead of focused evidence per angle.

Input: state["question"]
Output: state["themes"] (list of strings)
Fallback: if JSON parse fails, the original question is used as a single theme
Model: Llama3 via POST /api/generate, stream: false

■ Agent 2

Retriever Agent

For each theme, embeds the theme string using nomic-embed-text and runs FAISS ANN search returning the top-12 most semantically similar feedback items. Deduplicates across themes by ID so the same review is never cited twice. Each hit is formatted with source, rating, and date before being passed to the Synthesizer.

Input: state["themes"]
Output: state["formatted_context"], state["cited_ids"]
Format: [App Store, 1?, 2024-03-10] "The checkout crashes..."
Supports optional filters: source_filter, min_rating
Max unique items: 48 per query (4 themes × 12, deduplicated)

■ Agent 3

Synthesizer Agent

Receives all retrieved feedback pre-formatted with attribution tags and writes a structured insight report using Llama3. The prompt explicitly prohibits claims without evidence from the context and enforces a three-section structure: Executive Summary, Key Findings per theme with verbatim quotes, and Priority Recommendations.

Prompt Constraints

Use ONLY the provided context � no external knowledge
Every finding must include a verbatim quote with [Source, ?, Date]
If a theme has no relevant feedback, say so explicitly
Recommendations must be specific and ordered by impact

Output Structure

## Executive Summary
(2-3 sentences, business impact)

## Key Findings
### Theme: [name]
(findings + verbatim quotes)

## Priority Recommendations
1. (most impactful, specific)
2. ...
3. ...

Evaluation Framework

Three Custom LLM-as-Judge Metrics

All three metrics use Llama3 as the judge, prompted to return a JSON object with a float score 0.0�1.0 and one-sentence reasoning. The score is clamped server-side. Scores are stored in eval_results and linked to the insight_query_id so every report has a permanent quality certificate.

Metric 01

Faithfulness

Can every claim in the report be traced to an actual user quote in the retrieved feedback? Penalizes hallucinated product features, invented statistics, or fabricated sentiment not present in any retrieved item.

0.82 avg at 1K+ items

Metric 02

Relevance

Does the report directly and completely address the product manager's question? A report that accurately cites real feedback but ignores the actual question scores low even if every claim is grounded.

0.79 avg at 1K+ items

Metric 03 � Custom

Specificity

"Users are unhappy with checkout" scores near 0. "7 App Store reviewers (all 1?) cite checkout crashes on iOS in March" scores near 1. Penalizes vague generalizations regardless of accuracy, which standard RAGAS metrics miss entirely.

0.76 avg at 1K+ items

Overall Eval Score vs. Feedback Volume in FAISS

Metric	What It Measures	Judge Prompt Input	Avg Score (1K items)	Fails When
Faithfulness	All claims grounded in retrieved feedback	feedback_context + report	0.82	Llama3 adds external knowledge or invents statistics
Relevance	Report answers the product question asked	question + report	0.79	Report addresses adjacent topics but ignores the actual question
Specificity	Cited quotes vs. vague generalizations	report only	0.76	Report says "many users" without quotes, counts, or source attribution
Overall	Simple mean of all three	computed server-side	0.79	Any single metric dropping below 0.60

Specificity is the differentiating metric for product feedback use cases. In testing, Llama3 would occasionally produce summaries that were technically faithful (every general statement was loosely supportable from the feedback) and relevant (addressed the question) but contained zero verbatim quotes. These reports scored 0.82/0.79/0.31 on F/R/S. Without the specificity metric, they would have appeared high-quality. With it, the low overall score flags them for human review before they reach the product team.

API Reference

REST Endpoints

All endpoints return JSON. Interactive Swagger UI available at /docs after startup. The three routes map directly to the three product workflows: ingest new feedback, run an insight pipeline, and query evaluation history.

Method	Path	Description	Key Response Fields
POST	`/feedback/`	Ingest single item with source, rating, category, date	id, embedded
POST	`/feedback/bulk`	Ingest list of feedback items in one request	ingested (count)
POST	`/feedback/pipeline/run`	Embed all unindexed items and load into FAISS	processed, total_indexed
GET	`/feedback/`	List feedback with filters: source, category, min_rating	id, source, content, rating, embedded
GET	`/feedback/stats`	Total count, indexed count, avg rating, by-source breakdown	total_feedback, indexed_in_faiss, by_source[]
POST	`/insights/`	Run full agent pipeline, returns report + themes + eval in one call	report, themes, eval, feedback_count, latency_ms
GET	`/insights/`	List past insight queries with themes and latency	id, question, themes, latency_ms
GET	`/insights/{id}`	Full report with cited feedback IDs and attached eval result	report, cited_feedback_ids, eval
GET	`/eval/stats`	Aggregate eval scores across all insight queries	avg_faithfulness, avg_relevance, avg_specificity
GET	`/health`	Liveness check with FAISS item count	status, feedback_indexed

VectorStore: Filtered Search

async def search(
    self,
    vec: np.ndarray,
    k: int = 10,
    source_filter: str | None = None,
    min_rating: int | None = None,
) -> list[dict]:
    fetch_k = min(k * 4, self._index.ntotal)
    scores, indices = self._index.search(
        vec.reshape(1, -1), k=fetch_k
    )
    results: list[dict] = []
    for score, idx in zip(scores[0], indices[0]):
        item = self._items[idx]
        if source_filter and item.get("source") != source_filter:
            continue
        if min_rating and (item.get("rating") or 0) < min_rating:
            continue
        results.append({**item, "score": float(score)})
        if len(results) >= k:
            break
    return results

Ingestion Pipeline

async def run_ingestion_pipeline(
    store: VectorStore
) -> dict:
    processed = 0
    async with AsyncSessionLocal() as db:
        while True:
            result = await db.execute(
                select(FeedbackItem)
                .where(FeedbackItem.embedded == False)
                .limit(BATCH_SIZE)
            )
            batch = result.scalars().all()
            if not batch:
                break
            await index_items(
                [{"id": f.id, "content": f.content,
                  "source": f.source, "rating": f.rating,
                  "source_date": f.source_date.isoformat()
                  if f.source_date else None}
                 for f in batch], store
            )
            await db.execute(
                update(FeedbackItem)
                .where(FeedbackItem.id.in_(
                    [f.id for f in batch]
                )).values(embedded=True)
            )
            await db.commit()
            processed += len(batch)
    return {"processed": processed}

Tech Stack

Infrastructure and Design Decisions

Every technology choice was driven by the constraint of running a full AI stack with zero cloud API dependencies. This shapes everything from the embedding model to how FAISS state is managed across restarts.

AI Layer

Llama3 + nomic-embed-text via Ollama

Both models run locally via Ollama's REST API (/api/generate, /api/embeddings). No API key, no usage cost, no data egress. nomic-embed-text produces 768-dimensional embeddings that outperform OpenAI text-embedding-ada-002 on semantic similarity benchmarks while running on CPU.

Vector Store

FAISS IndexFlatIP

Inner product on L2-normalized vectors equals cosine similarity. IndexFlatIP does exact search (no approximation error) which is appropriate at <100K items. All writes go through an asyncio.Lock to prevent concurrent FAISS mutations from multiple FastAPI workers. Metadata stored in a parallel Python list indexed to match FAISS row numbers.

Database

PostgreSQL + SQLAlchemy 2.0 async

Three tables: feedback_items (source, rating, embedded flag), insight_queries (report, themes, cited IDs), eval_results (three scores per query). asyncpg driver with async_sessionmaker. Tables auto-created at startup via create_all. The embedded flag is the key design: it makes FAISS fully reconstructible from PostgreSQL.

API Layer

FastAPI with lifespan DI

The VectorStore singleton is initialized in the lifespan context manager and injected via FastAPI's Depends() system � never a global variable. The ingestion pipeline runs automatically at startup to re-index any backlog. All LLM and DB calls are fully async using httpx and asyncpg, so the server never blocks a thread.

Infrastructure

Docker + GitHub Actions CI

docker-compose brings up PostgreSQL 16 and the FastAPI app together, with a healthcheck condition ensuring the DB is ready before the app starts. GitHub Actions runs ruff linting, pytest, and a Docker build on every push to main and develop branches. The CI PostgreSQL service matches the production config exactly.

Agent Pattern

BaseAgent ABC + shared state dict

Each agent implements async run(state: dict) -> dict. The orchestrator calls them sequentially, each transforming the state. No framework dependency (no LangChain, no CrewAI) � the pattern is intentionally minimal so agents are individually unit-testable. Adding a 4th agent (e.g., a Critic) means writing one class and appending it to the pipeline list.

Why not LangChain or CrewAI? Both frameworks add significant abstraction overhead for what is a three-step sequential pipeline. The custom BaseAgent pattern keeps the entire orchestration in under 30 lines of code, makes each agent independently testable, and avoids the framework version-pinning issues that break LangChain pipelines on minor releases. The same pattern scales to 10+ agents without increasing complexity.

The over-fetch strategy in FAISS search is critical. Fetching 4×k results before applying source/rating filters ensures that filtered queries still return k results even when many items don't match the filter. Without this, a min_rating=1 filter on a corpus that is 80% positive reviews would return near-empty results despite having relevant 1-star items indexed.