Filling out forms might seem trivial for a human, but for a software agent it represents a major technical challenge. When the form is a pure image — no OCR, no DOM, no semantic labels — the difficulty multiplies. The agent must visually recognize fields, understand natural language instructions, retrieve personal or business information, and finally fill each blank with pixel‑perfect accuracy. This scenario appears frequently in digital onboarding, administrative document processing, tax filings, credit applications, or government platform registrations. Until now, most benchmarks for graphical agents focused on structured web environments, leaving aside the real world of scanned or photographed documents.
To fill that gap, a team of researchers has introduced FormGym, a benchmark specifically designed to evaluate intelligent agents in form filling from the purely visual domain. The corpus includes 432 fields distributed across 55 real documents, covering three tasks that require handling 236 attributes per simulated user. Initial results are revealing: state‑of‑the‑art Vision‑Language Agents (VLAs) barely achieve 1% accuracy in most cases, failing mainly on spatial field localization. Traditional GUI agents score between 10.6% and 68%, but at very high computational cost and latency. To improve this, the study introduces FieldFinder, an assistance tool that helps language models (LLMs) identify exactly where to place text on the form. With FieldFinder, all models improve, in the best case jumping from 2% to 56% accuracy.
These findings have direct implications for enterprise solutions. At Q2BSTUDIO, a company specialized in custom software, we know that automated form filling is a growing demand in sectors such as banking, insurance, logistics, and public administration. The ability of a system to process a scanned document and extract the necessary information without manual intervention saves hours of work and reduces errors. However, as FormGym shows, current technology still suffers from precision issues, especially in field localization. This is where complementary solutions come into play: domain‑specific language models, improved computer vision with convolutional neural networks, and the integration of tools like FieldFinder that act as a visual 'copilot' for the LLM.
From a technical perspective, the benchmark poses three progressive tasks: simple field filling, filling conditioned on business rules (e.g., if document type is passport, the nationality field must be the issuing country), and filling with external information that requires querying a database or an API. This last task accurately mirrors real‑world scenarios where the agent must retrieve customer, product or order data from CRM or ERP systems. For a company like Q2BSTUDIO, which offers AI and custom AI agents, these challenges are everyday work. Building an agent that reads a scanned form, interprets the prompt 'Enter your policy number' and then queries a back‑end system to obtain that data requires orchestrating multiple capabilities: assisted OCR, semantic reasoning, API integration, and a user interface that shows progress to the operator.
The study also highlights the importance of cybersecurity in such processes. When an agent handles sensitive data — names, addresses, tax identification numbers — transmission and storage must comply with regulations like GDPR or local equivalents. A poorly designed solution could expose confidential information if encryption, access control and auditing are not implemented. At Q2BSTUDIO we embed cybersecurity in every layer of development, from agent authentication to cloud service communication. Indeed, cloud AWS/Azure infrastructure is a key enabler: it allows scaling form‑filling processes with high availability and hosting AI models that require powerful GPUs. Additionally, BI/Power BI services can consume the data extracted from forms to build performance dashboards, anomaly detection or compliance reports.
The cost and latency metrics mentioned in the FormGym experiments are another critical factor for enterprise adoption. An agent that takes 30 seconds to fill a single field and consumes expensive API credits is not viable for processing thousands of forms daily. That is why model optimization — via quantization, distillation, or pruning — is an area we address at Q2BSTUDIO with agile methodologies. We combine pre‑trained models with fine‑tuning on proprietary datasets and evaluate performance in real environments before going to production. The use of tools like FieldFinder shows that a larger model is not always needed; sometimes a well‑designed external assistant can compensate for the base model's weaknesses.
Furthermore, this research opens the door to more ambitious applications. If an agent can fill forms from images, it could also verify that the content is correct, correct errors, and even digitally sign the document. This connects directly with the trend of intelligent process automation (IPA), where software robots (RPA) empowered with AI manage complete flows from receiving a scan to archiving in a database. At Q2BSTUDIO we develop such modular solutions: an image acquisition module, a recognition engine, a business‑rules manager, and a connector to legacy systems. Experience with benchmarks like FormGym helps us calibrate expectations and choose the most suitable architecture for each client.
In summary, FormGym is not just an academic benchmark; it is a realistic snapshot of the state of the art in an area that affects dozens of business processes. The results underscore that there is still a long way to go, especially in visual localization and computational efficiency. Yet tools like FieldFinder prove that significant advances can be achieved by combining language models with specialized modules. For companies looking to digitize their document management processes, the most sensible strategy is to rely on technology partners with multidisciplinary expertise: custom software, artificial intelligence, cybersecurity, and cloud. At Q2BSTUDIO we offer exactly that integrated vision, helping our clients turn a technical challenge into a competitive advantage.
