The explosion of AI agents has transformed the way companies approach automation and decision-making. But with this capability comes a silent challenge: the unpredictability of performance. Unlike traditional applications, where a database index or heavy query accounts for the slowness, in an LLM-based system a single user request can chain multiple external API calls, vector database searches, prompt formatting, and streaming responses. If a customer complains that their virtual assistant is slow to respond, it's not enough to just look at the server's CPU usage. You need to know which specific model was most time-consuming, how many tokens were spent, and whether context retrieval was the bottleneck. The answer to this need for deep visibility is OpenTelemetry applied to the world of AI.
At Q2BSTUDIO we have been helping companies build custom applications for years that solve real problems. Now, with the emergence of generative artificial intelligence, we have seen that observability is no longer a luxury but a requirement to maintain cost control and service quality. Our team integrates custom telemetry solutions into AI agent pipelines, allowing our customers to monitor not only the status of the server, but every logical step of the conversation with the model.
The key is not to settle for generic self-instrumentation that only reports that an HTTP request was made. To get really useful metrics—such as the number of input and output tokens, the model name, or the exact latency of vector search—you need to write explicit semantic traces. This is achieved by creating a custom tracer with OpenTelemetry that defines a parent span for the full execution of the agent and child spans for each operation: the query to the vector database, the call to the LLM and, if you are working with streaming, the time to the first token. This gives us a detailed time map that can be visualized in tools such as SigNoz or Jaeger.
Let's imagine a real scenario: a company uses an AI agent to answer technical questions about its products. The client asks and the system, after searching a vector knowledge base, sends the context to GPT-4o. With the right instrumentation, we can see that vector search consumed 400 milliseconds, model generation 1.8 seconds, and 370 tokens were spent in total. If a request suddenly takes 5 seconds, the flame graph will instantly show us which step was lengthened: perhaps the vector database started to degrade due to lack of indexes, or the LLM provider had a one-off latency. Without these traces, we would be blindly engineering.
Beyond latency, cost is the other big headache. Each token has a price, and when you scale agents to hundreds or thousands of users, spending can skyrocket. By attaching attributes such as llm.total_tokens and llm.model_name to spans, we can build dashboards in Business Intelligence tools such as Power BI or directly on top of ClickHouse, grouping by model and adding tokens. This becomes an executive dashboard that shows in real time how much we are spending for each version of GPT or Claude. It is a business intelligence service applied to the operation of AI.
Another critical aspect that we discover in our custom software projects is the need not to oversize traceability in asynchronous flows. If the agent returns the token-by-token response via streaming, wrapping the entire loop in a span artificially inflates the latency, because it includes the client's network speed. The correct thing to do is to measure the time from when the request is sent until the first byte arrives – the famous Time-To-First-Token – and then let the rest of the transmission flow without being timed as part of the main span. In addition, in production environments, a BatchSpanProcessor, not the SimpleSpanProcessor, should always be used to prevent the application from waiting for the trace collector to confirm receipt of data, which would degrade end-user performance.
The integration of this telemetry with the cloud infrastructure is natural. Many of our customers deploy their agents on AWS and Azure cloud services, and using OpenTelemetry allows you to centralize traces regardless of vendor. Combined with cybersecurity policies that prevent the leakage of sensitive data through the attributes of the spans, we achieve a robust and auditable system.
At Q2BSTUDIO we believe that AI for business should be measurable and governable. That's why we offer consulting and development of these solutions, helping organizations implement AI agents with full traceability. If you want to learn more about how to implement your own pipelines, we invite you to learn about our focus on artificial intelligence and observability and how we apply this knowledge to real projects. Transparency in cost and performance is no longer an option – it's the foundation for scaling AI with confidence.
