When a technology company like Zillow states that the real challenge is not data but context, it forces a rethinking of how to truly measure return on investment in artificial intelligence. The company, involved in approximately 80% of U.S. real estate transactions, has been using machine learning for decades. Yet its engineering executive, Toby Roberts, made clear at a recent event that AI ROI is only tangible if baseline metrics are established before launching any model. This lesson resonates strongly with any organization aiming to adopt intelligent agents without wasting compute or losing critical information.
The core idea is simple: without a performance baseline, any improvement attributed to AI can be a statistical illusion. Zillow implemented DORA metrics years before its current automation wave, allowing it to precisely measure a 40% increase in code shipped to production thanks to its coding assistants. That data point would not have been possible without a prior measurement framework. For most companies, building that framework involves auditing processes, identifying bottlenecks, and defining key indicators before touching a single language model.
This is where the value of having a technology partner that understands both infrastructure and business strategy becomes evident. At Q2BSTUDIO, we have helped dozens of organizations design that baseline from scratch, combining custom software with deep knowledge of software lifecycles. It is not just about installing a chatbot; it is about orchestrating cloud services, protecting data with advanced cybersecurity, and building a governance model that allows scaling artificial intelligence without losing control.
Zillow's case also illustrates the importance of not delegating context to a single chat interface. The company chose to build its own persistent context layer, capable of following the customer across different touchpoints over months. That decision reflects an uncomfortable truth: models alone are not enough. They need an architecture that remembers where the user left off, what permissions they have, and what sensitive information must be protected. And that, as experts pointed out, is much harder than cleaning data.
Instead of chaining customers to a single language model, Zillow opted for a harness of small, specialized models trained for specific tasks. This approach, already used by many leading companies, reduces token consumption and improves latency. To replicate it, you need a platform that centralizes integrations and prevents each department from building its own connectors. This is where the cloud AWS/Azure ecosystem offers competitive advantages, allowing the deployment of multiple agents with unified security policies.
Another key takeaway from Zillow's approach is that ROI measurement cannot be limited to the development phase. It must extend to daily operations. Glean co-founder Arvind Jain explained how intelligent model routing can cut token consumption by half, simply by sending simpler tasks to smaller, cheaper models. Additionally, precomputed context prevents each agent from rebuilding the thread from scratch, a saving that often goes unnoticed in AI budgets.
If a company wants to implement AI agents with measurable returns, it should start by auditing current processes, establishing productivity and cost indicators, and then designing an architecture that separates context from the underlying model. At Q2BSTUDIO we work precisely along those lines: we help organizations build the context layer their business needs, whether through integrations with legacy systems, cloud migrations, or the development of dashboards with BI/Power BI to monitor agent performance in real time.
Equally important is cybersecurity. Zillow added strict rules and additional compliance checks for its most sensitive data, even after implementing a permissions-aware platform. That prudence is fully applicable to regulated sectors such as banking, healthcare, or fintech. When an AI agent accesses confidential information, trust in the system depends on that access being controlled, audited, and reversible.
Ultimately, Zillow's story is not an isolated success case but a practical guide for any company looking to obtain real ROI from its artificial intelligence investments. Measuring before building, centralizing context, specializing models, and hardening security are the pillars of a strategy that transcends headlines. And on that path, having a team that masters both the technical and business sides makes the difference between a costly experiment and a sustainable transformation.
