rsi.watch

Xiangyu Peng et al. (7 authors) · 2026-10-24

MTA-Zero: Self-Improving Multimodal Deep Search via Online QA-Solver Co-training

“Across six multimodal deep-search benchmarks, MTA-Zero improves a Qwen3-VL-8B Solver from 36.91% to 51.58% mean accuracy and achieves performance comparable to GPT-5 under the same agent workflow.”
Who
Xiangyu Peng et al. (7 authors)
Date
2026-10-24
Evidence
Very low certainty Starts at Low: the developer’s own numbers.
  • ↓ risk of bias. FVQA's training split supplies 300 of the seed questions and its test split is one of the six benchmarks. The paper reports no check for overlap between generated training questions and benchmark items. “official FVQA test split (FVQA)”
  • ↓ imprecise. The paper reports no repeated training runs; evaluation uses 3 generation seeds, with no spread reported. The frozen and trained Generator ablation is reported once each on four benchmarks. “Results report mean accuracy (%) over 3 generation seeds.”
how we rate evidence
Replication
No rerun found. The paper links no code or weights. We found no rerun by another group. Searched up to 2026-10-28. all claims
Checked
2026-10-28, against the source page
Links
Source Paper

Also on the page

  • Two separate copies of Qwen3-VL-8B-Instruct are trained with GRPO for 50 iterations. The QA Generator starts from one-hop image questions (InfoSeek, FVQA-train, InfoVQA, OK-VQA), searches the web and builds questions "targeting a total depth of 5 hops", which a fixed harness checks. The Solver answers each question with tools in 8 rollouts and is trained on those rollouts.
  • The Generator's reward comes from how many of the 8 Solver rollouts answer correctly: zero when all or none do, highest at 4 of 8. On four benchmarks, the base Solver averages 35.87%. Trained against a frozen Generator it reaches 44.28%, and with the Generator trained too, 52.40%.
  • The six evaluation benchmarks are existing test sets, separate from the generated training questions. One is the FVQA test split, and FVQA's training split supplies 300 seeds; the paper reports no overlap check between generated questions and benchmark items. The authors ran their own model and the commercial baselines in the same agent workflow, mean of 3 runs: GPT-5 averages 51.98%, Gemini-3-Pro 54.28% and Claude 4.5 Sonnet 58.03%. The authors: "Although stronger commercial agents remain ahead overall". A Qwen3.6-27B Solver went from 46.88% to 57.70% on four benchmarks.
  • The authors' limits: comparisons with specialist agents are "not a cost-matched comparison"; their quality audit of generated questions "does not rule out every possible shortcut"; and "Exact replay of live retrieval and proprietary model-assisted checks may vary with web content and service versions".

In the self-improving AI catalog

  • MTA-Zero: what it changes, whether the change lasts, and how it was tested

← Ledger Report an error