Deploying large language model (LLM) applications to millions of users faces a critical bottleneck: inference cost and latency. While advances in hardware and model optimization continue, a complementary strategy that has gained traction is intelligent clustering of input queries. The idea is simple yet powerful: group similar requests, process only one representative per cluster, and reuse the output for the rest. However, this approach is only safe if we can guarantee that each element in the cluster is genuinely close to its representative, both in semantic terms and categorical attributes. Traditional clustering methods do not offer these guarantees at scale: they do not verify a minimal intra-cluster similarity, they do not ensure exact match of categorical attributes, and they often become intractable with tens of millions of samples. In this article we explore a two-stage solution that combines Mini-batch K-Means with a greedy selection of representatives inspired by the Johnson-Chvátal heuristic for Set Cover. This approach not only scales linearly in the number of samples when the number of initial clusters grows proportionally, but also provides per-sample quality guarantees, drastically reducing cost and latency in production. From a business perspective, such techniques enable launching personalized recommendation systems, conversational assistants or AI agents that were previously unfeasible due to high inference costs. At Q2BSTUDIO, as a software and technology development company, we integrate these strategies into custom software applications for our clients, combining efficient clustering with cloud infrastructure and cybersecurity measures.
The fundamental problem of scaling LLMs is not only computational cost but also latency when each request must go through a large model. Consider a persona-based recommendation system. Each user generates a textual query describing their situation or preference. If we had to send every query individually to the LLM, weekly costs could skyrocket to millions of dollars and response times would be unacceptable. The natural solution is to cluster queries. But naive clustering may group queries that appear similar but have subtle differences that alter the desired response. For instance, two users might ask 'recommend a sci-fi movie', but one wants recent releases and the other classics. If the representative only captures the generic part, the response can be wrong for a portion of the group. That is why we need guarantees that each element is within an alpha radius (in embedding space) of the representative, and that categorical attributes like gender, age or location match exactly.
The algorithm we propose, based on ideas from the reference paper (arXiv:2607.19704v1) but adapted to a business context, operates in two phases. In the first phase, Mini-batch K-Means is applied to the query embeddings with an initial number K of clusters. This provides a coarse grouping that scales to millions of samples thanks to batch processing. In the second phase, within each initial cluster, a subset of representatives is selected greedily such that every point in the cluster is within a maximum distance alpha from at least one representative, and all representatives share the same categorical attribute values with their members. This step is equivalent to solving a Set Cover problem over balls of radius alpha in embedding space, using the Johnson-Chvátal heuristic, which provides a polynomial-time approximation. The result is a set of representatives with coverage guarantees: each query assigned to a representative is similar to it within a measurable margin, and categorical attributes match exactly.
From a complexity standpoint, the algorithm has a time cost of O(nd + n^2 d / K) and a memory cost of O(nd + n^2 / K^2), where n is the number of samples, d the embedding dimension, and K the number of initial clusters. When K grows proportionally to n, the quadratic term linearizes, allowing scaling to 38 million customers as demonstrated in a real-world deployment. In that case, the method reduced inference cost and latency by a factor of 50 while preserving personalization and unblocking the production launch. Such results are not only technically impressive; they open the door to applications that were previously economically unviable.
For a company like Q2BSTUDIO, implementing this solution requires integrating several technological components. On one hand, an efficient embedding generation system is needed (e.g., using lightweight models or cloud embedding services). Then, the clustering and representative selection algorithm must run on a scalable environment, typically on cloud infrastructure like AWS or Azure. At Q2BSTUDIO we offer cloud services on AWS and Azure that can host massive data pipelines, worker orchestration and embedding storage. Moreover, cybersecurity is critical when handling data from millions of users: the assignment to representatives must ensure no sensitive data leaks, so we apply anonymization techniques and access control. Our cybersecurity and pentesting team audits these systems to comply with regulations like GDPR.
Artificial intelligence, and particularly AI agents, greatly benefit from this clustering-with-guarantees approach. For example, an agent answering customer support queries can process thousands of queries in parallel by grouping them by intent and context, using a representative to generate a base response that is then lightly adjusted for each member. This drastically reduces the number of LLM calls and enables real-time agent action. At Q2BSTUDIO we develop AI and intelligent agent solutions that incorporate these optimization algorithms, along with Business Intelligence (Power BI) dashboards to monitor response quality and cluster coverage. Integration with BI allows product teams to visualize which user groups are receiving personalized responses and where deviations occur, adjusting alpha parameters or clustering granularity.
In summary, the combination of efficient clustering with per-sample quality guarantees solves the LLM inference bottleneck at scale. It is not just a technical optimization; it is a business enabler that allows launching personalized AI products to millions of users at affordable costs. Companies like Q2BSTUDIO are prepared to implement these architectures, from developing custom software applications to managing cloud infrastructure, cybersecurity and data analytics. If your organization seeks to scale its AI applications without multiplying costs, this approach represents a solid and proven path.
