The advancement of digital assistants and automated agents has brought a recurring challenge to the table: how to make a machine not only execute tasks within a web interface, but also be able to explain step by step what it is doing, just as an expert human would do for a novice user. Until now, research on web agents has been divided into two separate fronts: on one hand, systems that execute actions on the DOM or the accessibility tree; on the other, models that generate guide text or tutorials. Both approaches ignore the reality that users interact with rendered screens, not with source code, and that a complex task rarely consists of a single click but rather a sequence of actions unfolding across multiple page states. This is where MAG (Multimodal Action and Guide) emerges — the first unified benchmark that simultaneously evaluates an agent's ability to complete a web task and to write a textual guide describing the process, all based on real screenshots. This approach not only represents a qualitative leap in measuring artificial intelligence applied to interfaces, but also opens concrete opportunities for companies that develop digital solutions, such as custom software or intelligent automation systems.
MAG proposes two grounding schemes on screenshots: Set-of-Mark, which selects visual elements via markers, and raw pixel coordinates. In this way, the agent must interpret the image exactly as a human sees it, without relying on hidden DOM metadata. The benchmark includes a complete harness covering everything from LLM-assisted annotation and human verification to training, evaluation in live environments, and joint metrics for actions and guides. The initial results are revealing: even the most powerful model currently available completes less than 40% of the tasks, leaving enormous room for innovation. For example, a supervised 9B parameter model barely achieves a 6.9% success rate, which rises to 13.2% after applying a GRPO training method augmented with expert trajectories. This improvement, while significant, is still insufficient for production environments, indicating that research on multimodal agents is still in an early phase.
The business relevance of MAG is immediate. Companies investing in AI and automation need tools that not only execute tasks but also generate clear documentation for user onboarding, regulatory compliance, or internal training. An agent that can perform a sequence of actions on a cloud platform and at the same time draft a visual step-by-step guide would drastically reduce support costs and accelerate the adoption of new features. At Q2BSTUDIO, as a software and technology development company, we observe this advance with great interest. Our team regularly works on projects integrating cloud AWS/Azure, cybersecurity, and BI/Power BI, and we know that a system's ability to self-document is a differentiating factor. For instance, imagine an agent that deploys an infrastructure on AWS, configures security policies, and also generates a guide for the operations team: that is exactly what MAG aims to evaluate and improve.
The benchmark also highlights the need for more robust multimodal models. Current agents, mainly text-based (DOM, accessibility tree), fail when faced with complex graphical interfaces with dynamic elements, pop-ups, or state changes that are not reflected in the underlying code. MAG forces the model to process the full image, including colors, icons, spatial layout, and visual content — something humans do naturally. This has direct implications for the development of automation processes: tasks that previously required fragile scripts on HTML selectors could be replaced by agents that understand the interface as an image, making them much more resilient to design changes. Moreover, generating guides in natural language makes the results understandable to any stakeholder, not just developers.
From a technical perspective, MAG introduces an innovative annotation methodology that combines the power of large language models (LLMs) with human precision. Instead of manually labeling each step, an LLM proposes initial annotations that are then verified and corrected by humans, reducing data preparation cost and time. The harness also includes a live evaluation environment where the agent interacts with real web pages and measures both task success (e.g., completing a purchase) and guide quality (clarity, completeness, step order). Joint metrics are an important advancement because an agent can complete a task but generate a confusing guide, or vice versa. MAG weights both aspects, better reflecting the real value for an end user.
Another relevant finding from the study is that training with GRPO (Group Relative Policy Optimization) and expert trajectories nearly doubles the success rate of a small model (from 6.9% to 13.2%). This suggests that the quality of training data — specifically, demonstrations of humans performing tasks — is more critical than model size. For companies like Q2BSTUDIO, which develop custom software with AI components, this lesson is key: investing in collecting real user trajectories can be more effective than simply scaling models. In fact, many of our automation projects include observation phases where we capture how employees perform repetitive tasks, to later train agents that replicate them. MAG offers a framework to formally validate such approaches.
The future of multimodal web agents lies in integrating vision and language into a single flow, and MAG is the first serious testbed to measure that progress. As models like GPT-4o, Gemini, or Claude improve in visual understanding, we will see agents capable of navigating complex interfaces with greater autonomy. However, the benchmark reveals that there is still a long way to go: the best current model does not reach 40% success, and guide quality is also improvable. This opens an opportunity space for startups and tech companies that want to specialize in the 'explanatory agent' layer, combining computer vision, natural language processing, and sequential reasoning. At Q2BSTUDIO, we closely follow these trends to integrate them into our AI, cloud, and cybersecurity solutions, always aiming to offer our clients tools that not only work but also train and empower their teams.
In summary, MAG is not just an academic benchmark: it is a reflection of where human-machine interaction is heading. Companies that adopt this paradigm early — agents that act and explain — will have a competitive advantage in terms of operational efficiency, user satisfaction, and regulatory compliance. From custom software development to BI dashboard implementation, to cloud process automation, the ability to generate automatic guides from real actions will transform the way we design and maintain software. At Q2BSTUDIO we are ready to accompany our clients in that transformation, bringing technical experience and strategic vision.
