Why Drug Name Confusion Breaks RAG Systems — And How to Fix It

Retrieval-augmented generation (RAG) is transforming how pharmaceutical companies search regulatory documents. But there's a critical failure mode that most benchmarks miss: confusable drug names.

When a regulatory professional asks about “Abacavir Sulfate,” how often does the retrieval system return information about the similarly-named “Abametapir” instead? We built a benchmark to find out—and discovered that standard embedding-based retrieval fails on nearly half of such queries.

The good news: with the right techniques, failure rates drop by 80%. Here's what we learned.

The Problem: Drug Names Are Designed to Sound Similar

Pharmaceutical naming follows conventions that make drugs within the same class sound alike. This helps clinicians, but creates a nightmare for retrieval systems. Consider these pairs:

Each pair shares the same first letter and differs by only 4-5 characters. To a traditional embedding model, these names look almost identical. But the regulatory requirements for each product are completely different. A retrieval error doesn't just return the wrong document—it returns information about an entirely different drug.

Building a Benchmark for Confusability

To systematically test this problem, we built a benchmark specifically designed to stress-test retrieval systems on confusable pharmaceutical names. The construction pipeline has four stages:

The key innovation is hard negative mining: for each document chunk, we identify the most semantically similar chunks from products with similar names. Then we generate questions that should retrieve the target chunk—not its confusable neighbors.

Question Generation & Validation

We used GPT-4o-mini to generate 2-3 discriminative questions per chunk, then validated each question using a Claude-based agent with access to search tools. The agent determined whether each question could be answered by exactly one chunk in the corpus.

The 84.2% exclusivity rate indicates that our question generation approach successfully creates queries that require precise retrieval. The 15.8% rejection rate caught questions where multiple products genuinely share the same requirement (e.g., identical dissolution testing protocols).

Sample Benchmark Queries

The benchmark contains natural-sounding questions that a regulatory professional might ask. Here are examples showing where standard retrieval succeeds and fails:

The third query fails because “dissolution testing requirements” uses nearly identical language across many product guidances. Without context about which product the chunk belongs to, the embedding can't distinguish between them.

Evaluation Results: Standard Retrieval Fails

We evaluated six retrieval strategies across three embedding models. The results reveal a stark divide between standard and context-enriched approaches:

The baseline results are sobering. Standard embedding retrieval achieves only 51.8% Recall@20—meaning nearly half of queries fail to retrieve the correct chunk in the top 20 results. BM25 (keyword search) performs even worse at 45.4%.

Why Context Enrichment Works

The dramatic improvement comes from a simple technique: prepending contextual information to each chunk before computing embeddings. Here's what that looks like in practice:

The contextual prefix anchors the embedding in product-specific space. When a user queries about “Abacavir Sulfate dissolution testing,” the contextualized chunk's embedding is much closer to the query than the raw chunk—even though the actual content is identical.

Model Comparison: Context Matters More Than Embeddings

We tested three embedding models to see if choice of model affects performance:

A striking finding: the choice of embedding model has minimal impact once context enrichment is applied. All three models achieve 89.9-90.3% Recall@20 with the best strategy. This suggests that context enrichment—not embedding quality—is the dominant factor for pharmaceutical retrieval.

Key Takeaways

Implications for RAG System Design

For teams building retrieval systems for pharmaceutical or other specialized domains, this benchmark highlights several design principles:

• Entity disambiguation is critical. Domain-specific retrieval often involves entities with similar names but different meanings. Standard embeddings conflate these; context enrichment separates them.
• Invest in context enrichment before embedding models. Switching from OpenAI to Gemini embeddings gains 3 percentage points. Adding context enrichment gains 37 points. The priorities are clear.
• Reranking is worth the latency cost. For high-stakes applications, the consistent 1-3 point improvement from reranking justifies the additional compute.
• Test on domain-specific hard negatives. Generic retrieval benchmarks (MTEB, BEIR) don't capture entity confusion. Domain-specific evaluation is necessary to understand real-world performance.

Limitations

Several limitations should be noted:

• Domain specificity: Results are from FDA pharmaceutical guidance documents. Performance on other regulatory corpora requires separate validation.
• Static corpus: The benchmark reflects a point-in-time snapshot. As FDA updates guidance documents, the benchmark would need refreshing.
• Levenshtein-based confusability: Our approach captures orthographic similarity but may miss phonetic or semantic confusability.

The remaining 10% failure rate represents cases where products have genuinely similar regulatory requirements—a fundamental ambiguity that retrieval alone cannot resolve.