In the competitive world of machine learning, optimizing large-scale language models has become both a technical and economic challenge. The need to reduce computational and memory costs has driven research into training with quantized weights, i.e., low-precision numerical representations such as BF16, FP8, or even FP4. However, this precision reduction often comes with a degradation in model accuracy, especially when high-precision copies are not maintained during optimization. This is where a new approach comes into play: M+Adam, an optimizer that combines additive and multiplicative updates to overcome the limitations of traditional techniques in low-precision environments.
The central idea of M+Adam stems from the observation that conventional optimizers, like Adam, use additive updates (adding a scaled gradient to the weights). Low precision implies coarse mantissa resolution, so when weights have large magnitudes, additive updates get rounded to zero, preventing any progress. Conversely, multiplicative updates (multiplying the weights by a factor) work well at large magnitudes but fail near zero or when the weight sign needs to change. These failure modes are complementary, and M+Adam exploits them by combining both update types: additive steps to handle sign changes and small magnitudes, and multiplicative steps to ensure progress at large magnitudes when additive updates are zeroed out by rounding.
Theoretical results show that M+Adam guarantees monotone descent under standard smoothness assumptions, and empirical tests with LLaMA models from 60M to 1B parameters, with Chinchilla budgets from 1x to 8x, and using only BF16, FP8, and FP4 master weights, consistently improve low-precision training accuracy. This has profound implications for the industry, enabling larger models to be scaled without skyrocketing costs.
In a business context, the ability to train AI models with high energy and memory efficiency is critical. Companies like Q2BSTUDIO integrate these advances into their AI solutions to offer clients faster and more affordable systems. Low-precision model optimization not only reduces the carbon footprint but also allows deploying artificial intelligence on edge devices or in cloud environments with limited resources.
But the application of M+Adam goes beyond pure training. In the development of custom software, computational efficiency translates into faster response times and lower latency. For example, in recommendation systems or virtual assistants based on AI agents, precision reduction allows complex models to run in real time without sacrificing quality. Q2BSTUDIO combines these techniques with its expertise in cybersecurity, cloud AWS/Azure, and Business Intelligence (Power BI) to create comprehensive solutions that protect data and optimize processes.
From a cloud perspective, AWS and Azure cloud services offer specialized hardware for low-precision training (such as GPUs with FP8 support). M+Adam benefits from these platforms, and Q2BSTUDIO helps companies migrate their AI workloads to the cloud, ensuring optimal performance and controlled costs. Similarly, in BI and Power BI, integrating AI models trained with low precision enables advanced predictions and analytics directly on business data, without massive infrastructure needs.
Cybersecurity is also strengthened: models trained with M+Adam can be deployed in sensitive environments where efficiency is key for real-time threat detection. Q2BSTUDIO offers pentesting and security services to ensure these models are not vulnerable to adversarial attacks, while also complying with data protection regulations.
In summary, M+Adam represents a significant advance in low-precision optimization, and its practical implementation is being led by companies that understand the synergy between research and commercial application. Q2BSTUDIO, with its focus on custom software development, artificial intelligence, cybersecurity, cloud, and BI, is perfectly positioned to help organizations adopt these innovations and transform their operations. The combination of additive and multiplicative updates not only improves accuracy but opens the door to a new generation of efficient and sustainable models.
