Google DeepMind Launches Multimodal Knowledge Agent Benchmark to Advance Visual Reasoning AI

Google DeepMind, in collaboration with Artificial Analysis, has introduced the Multimodal Knowledge Agent (MKA) benchmark, a novel evaluation framework designed to assess how AI agents perform on knowledge work tasks involving visually complex documents. Unlike traditional benchmarks focused purely on text, MKA incorporates documents like architectural blueprints, engineering schematics, reports, and spreadsheets, reflecting the multimodal nature of everyday professional workflows.

MKA’s evaluation set comprises 200 tasks across 10 verticals and 83 occupations, curated alongside domain experts including architects, biologists, underwriters, and data scientists. Each task simulates multi-step projects requiring three to eight hours for professionals to complete, pairing realistic prompts with multimodal input files. Outputs are measured against a detailed rubric emphasizing 75% factual accuracy and 25% visual design quality through 70 yes/no checks.

To grade AI outputs consistently, an agentic autorater built on a non-Google language model automatically inspects every file, rendering pages and evaluating spreadsheet formulas against the expert rubric. Domain expert reassessments confirmed 96% agreement with this automatic grading, establishing high reliability and objectivity in evaluations.

The MKA benchmark is hosted by Artificial Analysis, which keeps the main 200-task dataset private to prevent AI models from training on or searching these tasks, preserving result validity. A public subset of 10 tasks is openly available on Hugging Face to encourage community participation and transparency in testing.

Technical findings reveal large disparities in how frontier models approach MKA tasks, with efficiency ranging from 30 to 130 interaction turns per task. Results underscore the crucial role of strong visual perception capabilities for success, and confirm that models generally deteriorate significantly when visual inputs are withheld.

Among evaluated systems, Google’s Gemini 4 Carbon model ranked second in quality while operating at roughly six times lower cost than the top performer, demonstrating a strong balance between performance and efficiency. Gemini 4 Carbon is accessible through Google AI Studio’s Gemini API, enabling developers to integrate this advanced multimodal reasoning technology.

MKA represents a significant step forward in AI benchmarks by simulating authentic, visually complex knowledge work scenarios neglected by prior text-centric challenges. It provides an open, rigorous foundation for developing agents capable of synthesizing visual and textual information to assist professionals across various sectors.

However, the benchmark’s emphasis on specific task types and its current dependence on an autorater system indicate room for further expansion and refinement. Continued research will be needed to broaden vertical coverage and improve the interpretability and generalization of agents assessed with multimodal inputs.