Google's MKA benchmark tests AI agents on 200 real professional files; only 10 tasks are public for inspection

What the MKA benchmark actually measures

The Multimodal Knowledge Agent benchmark evaluates how AI agents handle visually complex professional work across 10 industry verticals and 83 occupations.¹ Unlike text-only benchmarks, MKA requires agents to analyze multi-file workspaces containing architectural drawings, engineering schematics, clinical scans, and other visual artifacts that professionals encounter daily.

Each of the 200 tasks represents a multi-step project that would take a seasoned professional three to eight hours to complete. Agents receive a realistic prompt and a bundle of multimodal input files, then must produce deliverables like reports, spreadsheets, slide decks, or marked-up drawings. An expert-written rubric of approximately 70 yes/no checks evaluates each submission, weighted 75% on factual accuracy and 25% on visual design.¹

Understanding the scoring: Climb vs Regression criteria

The benchmark's headline score is Climb Weighted Satisfaction Rate (WSR), which measures performance on the harder criteria that distinguish advanced capability from basic competence. According to Artificial Analysis, which operates the evaluation, criteria are labeled as either Climb (harder) or Regression (baseline).²

Climb WSR represents the mean weighted share of Climb rubric criteria satisfied per task, while Regression WSR measures baseline competence. This distinction matters because a model might perform well on basic requirements while struggling with advanced challenges. Google claims domain experts agreed with the automated judge's ratings 96% of the time when grading model outputs themselves.¹

What you can and cannot reproduce publicly

Only 10 showcase tasks are publicly available on Hugging Face for inspection and testing.³ These represent 10 unique domains and verticals, including:

Logistics & Mapping (GIS Analyst)
Construction (Architect)
Engineering (RF IC Design Engineer)
Science (Biochemist)
Finance (Investment Analyst)

Each public task includes metadata, the full prompt, input file list, and rubric criteria. However, the remaining 190 tasks remain private to prevent training data contamination and search-based shortcuts. This means you can examine the task format and test your own agents on the public samples, but you cannot reproduce the full benchmark evaluation that generates the published scores.¹

Interpreting benchmark costs and efficiency claims

Artificial Analysis reports cost per task figures that cover only calls to the model being tested, excluding judge calls.² This means the published costs don't represent the total expense of running a complete evaluation, let alone real workplace deployment.

Google's announcement highlights that models tested show "extreme differences in efficiency," using between 30 and 130 turns on average to complete each task.¹ The benchmark runs every task in the Stirrup harness with a limit of 250 turns, providing agents with a Linux sandbox, code execution, internet access, web search tools, and image-viewing capabilities.²

When comparing cost claims, note that Google's statement about Gemini 4 Carbon completing tasks at "6x lower cost on average compared to the top scoring model" refers specifically to benchmark-specific costs within this evaluation framework.¹

Legitimate comparisons and known limits

Legitimate comparisons using MKA scores should:

Distinguish Climb from Regression scores - Higher Climb WSR indicates better performance on challenging criteria
Recognize the 10/200 public/private split - Scores reflect performance on tasks you cannot inspect
Understand cost exclusions - Published costs omit judge calls and infrastructure
Consider task diversity - The benchmark spans 10 verticals but may not represent all professional domains

Google's technical report shows that "strong perception abilities are necessary to solve each task," and models "try and fail to compensate when vision is removed".¹ This validates the benchmark's focus on multimodal capability rather than text-only reasoning.

The benchmark represents a step toward evaluating AI agents on realistic professional work, but its private nature limits independent verification, and its cost metrics don't translate directly to workplace economics.

Share this article

Related News

ReproBridge preprint reports lower API costs for reusable ML artifacts

OpenAI Agents SDK preview separates local profiles from portable checkpoints

Workflow adds atomic starts

Google's MKA benchmark tests AI agents on 200 real professional files; only 10 tasks are public for inspection

What the MKA benchmark actually measures

Understanding the scoring: Climb vs Regression criteria

What you can and cannot reproduce publicly

Interpreting benchmark costs and efficiency claims

Legitimate comparisons and known limits

Sources

Tags

Google's MKA benchmark tests AI agents on 200 real professional files; only 10 tasks are public for inspection

What the MKA benchmark actually measures

Understanding the scoring: Climb vs Regression criteria

What you can and cannot reproduce publicly

Interpreting benchmark costs and efficiency claims

Legitimate comparisons and known limits

Sources

Tags