Skip to main content

Testing RAG Sources

Before relying on a RAG source in production, verify the connection and retrieval quality. This guide covers the test tools available and how to troubleshoot common issues.

Test Connection Button​

In the RAG Editor (or RAG Sources list), click Test Connection. This:

  1. Attempts to connect to your vector database using the configured credentials
  2. Sends a test search query (embedding of the word "test")
  3. Reports:

Success Response​

✅ Connection successful

Index: product-docs
Documents indexed: 1,247
Top result for "test": "Testing your integration..." (score: 0.71)
Dimensions: 1536 ✓
Metric: cosine ✓

Failure Response​

❌ Connection failed

Error: 401 Unauthorized — Invalid API key
Check: Your Pinecone API key in the RAG source configuration.

Verifying Retrieval in Agent Test Console​

The Test Connection button only checks connectivity. To verify that retrieval quality is good, test via the agent:

  1. Open the Agent Editor for an agent that uses your RAG source.
  2. In the Test Console (right column), type a question you know the answer to from your indexed documents.
  3. Review the response — does it contain specific information from your docs?

Good sign: The agent mentions specific details, names, or versions that could only come from your indexed content.

Bad sign: The agent gives a generic answer that could come from LLM training data.

Debugging Retrieval Issues​

Issue: Agent ignores indexed content​

Symptom: Agent answers generically despite having relevant content in the RAG source.

Check 1: Wrong embedding model for indexing Your indexed vectors were created with a different model than TARX's text-embedding-3-small. Vectors from different models are incompatible — cosine similarity between them is meaningless.

Fix: Re-index using text-embedding-3-small with 1536 dimensions.

Check 2: Score threshold too high If all retrieved results have scores below your threshold, they're filtered out before injection.

Fix: Lower the score threshold (e.g., 0.7 → 0.5) and re-test.

Check 3: Wrong index / collection name TARX is querying an empty or incorrect index.

Fix: Verify the exact index name in your vector DB console and update the RAG source config.

Check 4: System prompt not instructing use of context The LLM may be ignoring the injected context if not guided.

Fix: Add to the system prompt:

Retrieved context from our documentation is provided at the start of each 
conversation. Use it as your primary reference. Prefer retrieved content 
over your training knowledge when they differ.

Issue: Retrieved content is irrelevant​

Symptom: Test Connection succeeds and content is retrieved, but it's not related to the question.

Check 1: Poor chunking Chunks are too large (diluted relevance) or split at wrong points (context is lost).

Fix: Re-index with smaller chunks (300-400 tokens) and better boundaries (paragraph splits).

Check 2: Low-quality source content The indexed content uses different terminology than users' queries. Example: docs say "Single Sign-On" but users ask about "SSO".

Fix:

  • Add keyword variants to document metadata
  • Pre-process documents to normalize terminology
  • Consider using Azure AI Search's semantic ranking to handle vocabulary mismatch

Check 3: Score threshold too low Results with scores of 0.5-0.6 may be semantic matches but not actually relevant.

Fix: Increase threshold (0.7 → 0.75) to filter out weak matches.

Issue: 401 Unauthorized​

Cause: API key is wrong, expired, or has insufficient permissions.

Fix per provider:

  • Pinecone: Check API key in Pinecone Console → API Keys. Ensure it's for the correct environment.
  • Azure AI Search: Use Admin key (not query key) if getting 403 on write operations. For read-only, query key is sufficient.
  • Weaviate Cloud: API key from Weaviate Cloud Console → Cluster Details → API Keys.
  • Qdrant: Verify the key exists in Qdrant Cloud Console or your self-hosted config.
  • Supabase: Must use service_role key, not anon key.

Issue: 404 Not Found​

Cause: The index, collection, or class name doesn't exist.

Fix:

  1. Open your vector DB console.
  2. Copy the exact index/collection name.
  3. Update the RAG source config with the exact string (case-sensitive for most providers).

Issue: Dimension mismatch​

Symptom: Error like "Expected dimension 768, got 1536" or similar.

Cause: Your index was created with different dimensions than the 1536 TARX uses.

Fix: Re-create your index with 1536 dimensions and re-index your documents.

Issue: Empty results​

Symptom: Connection succeeds but document count shows 0, or all queries return no results.

Check 1: Index is empty — you haven't indexed any documents yet. Check 2: Wrong namespace (Pinecone) — your documents are in a different namespace. Check 3: Vectors stored under a different field name than configured.

Document Count Check​

A quick sanity check: the Test Connection report shows the document count. Verify this matches what you expect based on your indexing process.

Document CountWhat It Means
0Empty index — no documents indexed
< 10Very sparse — likely a test index or indexing issue
10-1000Small knowledge base
1000-100KTypical documentation or KB
100K+Large knowledge base

Advanced: Inspecting Retrieved Chunks​

To see exactly what chunks are being retrieved (useful for debugging quality):

In the Executions page, when an agent with a RAG source has run:

  1. Open the execution detail.
  2. Click the agent node to expand it.
  3. Look for the RAG Retrieval section — shows the top retrieved chunks and their scores.

This lets you see exactly what context was injected into the LLM's call, which is the most direct way to verify retrieval quality.

Last updated on