What an Agent Actually Gets From Several Providers
- Our agent answered 81.3% of 905 questions correctly with several providers at once, versus 77.0% with the best single provider.
- In hindsight, 89.7% of questions were answered correctly by at least one of 11 separate runs, one per provider. Several providers at once closed about a third of the gap between 77.0% and that 89.7% hindsight score.
- Using several providers required roughly two to four times as much text processing per correct answer.
Every setup used the same agent and 905 questions. The best single provider was selected after testing.
Each question had 11 separate runs, one per provider. All 11 runs were correct on 32.5% of questions.
Giving our search agent several providers produced 39 more correct answers than the best single provider. The agent, which searches, reads results and writes answers, attempted the same 905 questions in each setup.
Correct answers across separate runs
We ran the agent separately with each of 11 providers. For any chosen group of providers, the hindsight score counts a question as correct if at least one separate run answered correctly. Across all 11 providers, the hindsight score was 89.7%. That gives each question 11 separate runs, and only one of them has to be right. With several providers at once, the agent makes one run per question, so it need not reproduce those separate successes.
In Fig. 1, the grey bar is the best single provider at 77.0%, and the orange bar is the hindsight score for all 11 providers at 89.7%. For the best two or three providers, picked after testing, the hindsight scores fall in between, at 84.1% and 86.4%. For 8.4% of questions, only one to three of the 11 runs answered correctly (Fig. 2).
The gain from several providers
When each search went to several providers at once, the agent answered 81.3% correctly, versus 77.0% with the best single provider. That 4.3-percentage-point gain closed about a third of the gap to the 89.7% hindsight score.
More results, more text to process
The extra answers came with roughly two to four times as much text processing per correct answer as the six highest-scoring individual providers. More search results made a bigger reading pile, but the runs don't establish how much that extra material helped or hindered the agent.
The agent also made fewer searches: 3.6 per question, versus 4.3 to 5.0 with those six providers. Earlier stopping is one possible explanation; we haven't tested that explanation directly.
So several providers at once gave our agent 39 more correct answers out of 905 than the best single provider, which we could only pick after testing. Each correct answer took roughly two to four times as much text to process. Whether that's worth it depends on what a wrong answer costs you.
Methods and setup
Questions
The 905 questions cover papers, code, companies, finance, filings, the general web and Wikipedia. Reference answers were recorded beforehand. We withhold 266 questions about posts on hard-to-search platforms because the questions can identify private people.
Agent
In October, opencode ran each question with each provider and with Telem sending searches to 4.7 providers on average. Runs allowed 120 seconds per attempt and up to three attempts; unfinished runs counted as wrong. These findings describe one agent and model.
Judge
One judge agent checked final answers against the references. Rejudging later in October accepted valid alternative answers and code solutions. Answers that another judge agent had already checked one at a time kept their verdicts. Scores rose by 1.5 to 2.9 percentage points across setups. Text-processing ratios use the original judgments; rejudging leaves the approximate two-to-four-times ratio intact.
Measures
The correct-answer rate is the share of questions the judge marked correct. A percentage-point gain subtracts one rate from another.
With several providers at once, the agent answered 736 questions correctly: .
The best single provider answered 697 correctly. The gain was percentage points; gains over the next five providers were 4.6 to 6.3 points.
Across all 11 providers, the hindsight score counts 812 questions answered correctly by at least one separate run. All terms below are correct-answer percentages.
Using the exact counts gives , about a third. The 95% uncertainty range for the 4.3-point gain was 2.3 to 6.3 points, calculated from differences on the same questions.
For text processing, we count input and output tokens, pieces of model text, across all runs and divide by correct answers. The comparison uses the six highest-scoring single providers.
September check
With an earlier model on 516 different questions, ten providers per search gave 77.5% correct versus 73.6% for the best single provider: +3.9 points, with a 95% range of 0.7 to 7.0. Results were cut at roughly 50,000 characters; later searches exposed only three to five providers' results.
Configuration
Providers used Telem's recorded defaults. Telem is a customer of all 11.
| Provider | Setting we used |
|---|---|
| brave | Web search, defaults |
| ceramic | Defaults; descriptions up to 1,000 characters |
| exa | Search type auto (default) |
| firecrawl | Search, defaults |
| parallel | Mode turbo (the default then) |
| perplexity | Search context low (default) |
| seltz | Tier pro; no scope set (recorded news-only default) |
| serper | Defaults |
| tavily | Search depth basic |
| tinyfish | Defaults |
| you.com | Older search interface (GET) |
| All | Called through Telem; summaries cut at 1,000 characters; no location |
Providers: brave, ceramic, exa, firecrawl, parallel, perplexity, seltz, serper, tavily, tinyfish, you.com.
Telem.ai Team.