Google’s Gemini 4 Carbon Leads New Visual AI Benchmark, Cutting Cost Sixfold

Professionals across architecture, engineering, finance and biology spend hours poring over blueprints, schematics and data tables—tasks that are visual by nature. Traditional AI benchmarks have focused on pure text, leaving a gap in measuring how well agents can understand and act on complex layouts. Google’s new Multimodal Knowledge Agent (MKA) benchmark fills that gap by testing agents on realistic, visually‑rich work packages.

At the same time, Google has rolled out Gemini 4 Carbon, a model positioned to excel on this benchmark. By pairing strong perception with cost‑effective inference, Gemini 4 Carbon promises to handle the same multi‑step, multimodal projects that would otherwise require several hours of human effort, marking a shift toward AI that can truly replace visual‑heavy knowledge work.

Source: Google Gemini Blog: Introducing the Multimodal Knowledge Agent benchmark ↗

The MKA benchmark was built with domain experts from ten verticals and 83 occupations, producing 200 multi‑step tasks that mirror three‑to‑eight‑hour professional projects. Each task bundles a realistic prompt, a set of multimodal files and a rubric of roughly 70 yes/no checks weighted toward factual accuracy and visual design. All models run in the same open‑source agent harness, Stirrup, with identical tools and a fixed turn budget.

When evaluated, Gemini 4 Carbon ranked second in quality among the tested models and delivered results at about six times lower cost than the top‑scoring competitor. The benchmark also revealed stark efficiency differences—models used between 30 and 130 interaction turns per task—while expert graders agreed with the automatic autorater 96 % of the time, underscoring both the challenge and the progress.

Source: Google Gemini Blog: Introducing the Multimodal Knowledge Agent benchmark ↗

MKA’s 200‑task suite is kept private to prevent leakage into training data, meaning the full set isn’t publicly available for independent replication. The benchmark currently covers only the selected verticals and occupations, so performance on other visual domains remains unknown. Access to Gemini 4 Carbon requires using the Gemini API, which entails usage costs that may be prohibitive for small teams or hobbyists.

Even the best‑performing model shows wide efficiency variance; some tasks still need over a hundred turns, indicating that prompt engineering and tool integration remain critical. The study also notes that when vision is removed, models struggle to compensate, highlighting that strong perception is still a prerequisite for many visual tasks. These factors temper expectations about a one‑size‑fits‑all solution.

Developers interested in visual AI can start by signing up for the Gemini API in Google AI Studio and running Gemini 4 Carbon on their own workflows. Google also released ten public MKA tasks on Hugging Face, allowing anyone to inspect the format and benchmark their own agents against a shared baseline.

Beyond testing, teams should identify specific visual‑heavy processes in their organization—such as reviewing architectural drawings or assembling data‑rich reports—and pilot Gemini 4 Carbon on those use cases. Monitoring the forthcoming technical report will provide deeper insight into model behavior and help refine prompt strategies for optimal efficiency.