RAG vs fine-tuning vs long context for a customer-support assistant
Answer quality ↔ Freshness
Context
Answer support questions from 4,000 help-centre articles that change weekly.
Context
We're building an assistant that answers customer questions from our help centre: about 4,000 articles, roughly 6M tokens, edited every week by the support team. Volume is ~30K questions a day.
Options on the table
A: Retrieval-augmented generation. Chunk and embed the articles, retrieve the top matches per question, answer with citations.
B: Fine-tune a smaller model on the articles plus 18 months of resolved tickets.
C: Long context. Put the relevant product area's articles (100–200K tokens) straight into the prompt and let prompt caching absorb the cost.
What I'm unsure about
Long context keeps getting cheaper and removes a whole pipeline. Is RAG still the default in late 2026, or is it becoming the thing we over-engineer?
Constraints
- Corpus
- 4,000 articles, ~6M tokens
- Updates
- Weekly
- Volume
- 30K questions/day
- Team
- 2 ML engineers
RAGvsFine-tuningvsLong context
Community verdict
10 engineers · 8 opinions
With these constraints, what would you choose?
One choice per engineer. You can change it any time.
Cost per answer →
Trade-offs
Dimensions
- Freshness
- 5RAG scores 5 of 51Fine-tuning scores 1 of 54Long context scores 4 of 5
- Answer quality
- 4RAG scores 4 of 53Fine-tuning scores 3 of 5
Community discussion
8 comments
Have you made this decision in production? Share your reasoning.
Sign in to commentFrom the compliance side: citations are non-negotiable for us. When an answer is wrong, support needs to see which article it came from and fix the source. Only RAG gives you that cleanly.
Fine-tuning is the wrong tool for knowledge that changes weekly. Every article edit becomes a training run, and the model can still confidently cite last month's pricing. Fine-tune for tone and format if you must, on top of RAG.
Long context is underrated for a v1. With prompt caching, a 150K-token product-area prompt costs a fraction of the uncached price on repeat questions, and you ship in days instead of a quarter.
Use it to learn what customers actually ask. Build RAG once you know.
Do the maths on 30K/day though. Even cached, 150K tokens per question adds up fast compared with 4K tokens of retrieved chunks.
RAG, but be honest about where the work is: ingestion and evaluation, not the vector database. Our first version used naive 512-token chunks and answered "how do I reset 2FA" with the article about resetting passwords.
What fixed it: chunking on headings, a hybrid BM25 + vector retriever, a reranker, and a 300-question golden set we run on every change.
Same experience. The golden set was the single best investment. Without it every prompt change is a vibe check.
How big did the golden set need to be before it caught real regressions?
Around 200 questions, stratified by product area. Below 100 the noise drowned the signal.