Evaluation
Evaluate source-grounded agents on the quality of their evidence and the
usefulness of their recommendations.
Core Questions
- Did the agent find fresh, relevant sources?
- Did extraction return enough useful content?
- Are claims supported by source evidence?
- Are recommendations specific and actionable?
- Did the workflow stay within cost and latency budgets?
- Can a user inspect where the answer came from?
Suggested Scorecard
| Dimension |
What To Check |
| Freshness |
Sources reflect the requested time window |
| Relevance |
Results match the approved context |
| Coverage |
Prospect, industry, and competitor lanes are represented |
| Grounding |
Every important claim maps to a source |
| Specificity |
Recommendations name concrete actions |
| Source diversity |
The brief does not depend on one domain |
| Cost |
Run stays within search, extraction, and token budgets |
| Latency |
User sees progress and receives output fast enough |
| Robustness |
Partial failures do not break the whole run |
Regression Set
Keep a small fixture set for repeatable tests:
- a known company profile
- expected competitors
- representative search results
- extracted page samples
- one high-quality target brief
- examples of weak or irrelevant sources
Use the fixture set to test prompt changes, schema edits, and UI changes without
spending provider credits.
Citation Validation
For every generated brief, validate:
- cited URLs appear in the evidence packet
- source titles are preserved
- no final section cites an invented source
- unsupported claims are either removed or marked uncertain
Human Review
For strategy-style outputs, a human reviewer should ask:
- Would I act on this today?
- Is the recommendation grounded in current evidence?
- Is the source credible enough for the action?
- Did the agent miss an obvious source category?
- Is the brief concise enough for the target workflow?