AI News

Google's MKA benchmark tests AI agents on 200 real professional files; only 10 tasks are public for inspection

Google DeepMind's new Multimodal Knowledge Agent benchmark evaluates AI agents on 200 private tasks requiring visual analysis of complex professional files like architectural blueprints and engineering schematics. While the benchmark's headline scores are based on these private evaluations, only 10 sample tasks are publicly available for inspection. The benchmark uses weighted criteria where harder 'Climb' criteria measure advanced performance, and costs reported by Artificial Analysis exclude judge calls, limiting direct workplace cost comparisons.