When a Search Misses, Another Provider Often Has the Answer
- A retry, the agent's next search after a miss, found the answer 11.3% of the time on the original provider, versus 18.9% on one other provider, on average.
- Switching to a provider in a different family, a group with similar results, added 9.0 percentage points. After allowing for some providers doing better on ordinary searches anyway, the remaining gain was too uncertain to establish.
- At least one of the other 11 providers found the answer for 52.4% of retries.
All three bars count the same 292 retries. The blue bar is a switch to one other provider, averaged over the other 11. The orange bar counts a retry if any of the 11 had the known answer in its top five.
Each row is one provider and the retries written after its first search missed, 19 to 25 each, so each row is rough. Providers are not named. The best single other provider can change from row to row and was picked after seeing the results. In the bottom row, both dots sit at 28.0%.
Changing providers helped more than sending the same retry back to the provider that missed. We replayed 292 retries written by an agent, keeping the words fixed while changing the search provider.
The original provider found the answer in 11.3% of retries, and one other provider in 18.9% on average (the first two bars in Fig. 1). Each result list is like a reading list: another provider can recommend a useful page without changing the question.
Switching between provider families
In part 2, we grouped providers mainly by similar result lists. Switching within a family produced no average gain. Between families, the answer rate rose from 10.4% on the original provider to 19.4% on the other provider: a gain of 9.0 percentage points.
Some providers also performed better on ordinary searches. After accounting for those differences, the remaining gain was too uncertain to establish an additional benefit from switching families.
Where the remaining answers appeared
At least one of the other 11 providers found the answer for 52.4% of retries (the orange bar in Fig. 1). That number assumes you check all 11 and count the retry if any of them has the answer. One other provider found it 18.9% of the time on average.
For 28.4% of the same 292 retries, only one to three of the other 11 found the answer. For 47.6%, none of them did. We saw this on first searches too, in part 1: none of the 12 providers had the answer for 16% of them.
What's next
A failed retry could reflect the provider's results rather than a poorly worded search. For us at Telem, the next test is whether finding those extra pages helps the agent answer correctly. Our next post follows the full process, from the first search to the final answer.
Methods and setup
Retries
One agent wrote the retries in September. We replayed the searches in October, retaining 292 retries with usable original-provider results. The delay and single agent limit how broadly the findings apply.
Hit rate
A known answer is a page or value recorded beforehand. An automated rule counts a hit when the known answer appears in the first five results.
The original providers had 33 hits: 33 ÷ 292 × 100 ≈ 11.3%. A retry/provider combination pairs a retry with one other provider. Other providers had 599 hits across 3,165 evaluated combinations, giving 18.9%; unavailable results were excluded. At least one of the other 11 had a hit on 153 of 292 retries, giving 52.4%.
Gains between families
The Google-results family contains serper, firecrawl, tinyfish, tavily and serpapi; brave and you.com form another. Each remaining provider forms a separate family, following part 2's grouping.
Both rates use the same evaluated retry/provider combinations: 512 within families and 2,653 between families. Between families, 19.4% − 10.4% = 9.0 percentage points; within families, both rates were 16.4%.
Accounting for provider performance
We estimated the expected gain from providers' hit-rate differences on ordinary searches within the same question category. Between families, the observed 9.0-point gain minus the expected 5.2 points left 3.8 points; within families, 0.0 minus 0.6 left −0.6 points.
The 95% uncertainty ranges were 4.8 to 13.4 points for the observed gain between families, and −3.8 to 3.7 within families. After adjustment, the between-family range was −0.1 to 7.9, including zero. To estimate sampling uncertainty, we drew 1,000 new samples from the recorded questions, allowing repeats and keeping each question's comparisons together.
Differences by original provider
Across original providers, at least one of the other 11 found answers for 28% to 72% of retries; the best single other provider found 28% to 64%. The best other provider varied by original provider and was selected after testing. Each original provider contributed only 19 to 25 retries.
Configuration
All providers used Telem's recorded defaults. Telem is a customer of all 12.
| Provider | Setting we used |
|---|---|
| brave | Web search, defaults |
| ceramic | Defaults; descriptions up to 1,000 characters |
| exa | Search type auto (default) |
| firecrawl | Search, defaults |
| parallel | Mode turbo (the default then) |
| perplexity | Search context low (default) |
| seltz | Tier pro; no scope set; news-only in our October audit |
| serpapi | Google search, defaults |
| serper | Defaults |
| tavily | Search depth basic |
| tinyfish | Defaults |
| you.com | Older search interface (GET) |
| All | Called through Telem; top five results; summaries cut at 1,000 characters; no location |
Providers: brave, ceramic, exa, firecrawl, parallel, perplexity, seltz, serpapi, serper, tavily, tinyfish, you.com.
Telem.ai Team.