Protein contacts in attention: one-step alternative to the Jacobian

In the rapid advancement of artificial intelligence applied to computational biology, predicting the three-dimensional structure of proteins remains one of the most complex and strategic challenges. Language models trained on enormous volumes of sequences have demonstrated a surprising ability to capture relationships between residues, but the computational cost of extracting that information remains high. A recent line of research has revealed that the contact signal between amino acids not only resides in the output logits, but is naturally concentrated in a very small subset of attention heads. This finding makes it possible to replace the costly categorical Jacobian method —which required multiple forward passes— with a one-step approximation, selecting a few relevant heads from a small sample of labeled proteins. Experimental results show that even an unweighted average of those heads outperforms the Jacobian in large bidirectional models, with a significant improvement in scenarios where contamination from pretraining data is removed. The methodology opens the door to more agile applications in drug discovery and protein engineering, where inference speed is critical. From a business perspective, integrating these techniques into workflows of AI for businesses allows reducing operational costs and accelerating experimental iterations. Furthermore, the generalization to architectures without masked language heads —through a hidden representation of the Jacobian— extends the scope to causal models, although the results suggest that bidirectionality in pretraining is key for attention to encode pair structure. For organizations seeking to implement cutting-edge solutions, combining these advances with custom applications makes it possible to scale contact prediction to large genomic databases. The company Q2BSTUDIO develops platforms that integrate these artificial intelligence capabilities with cloud services, offering everything from cloud services aws and azure cloud services aws and azure to Power BI dashboards for monitoring model performance. In an environment where cybersecurity of research data is a priority, our pentesting and infrastructure protection solutions ensure the integrity of the pipeline. Thus, the leap from the laboratory to production becomes viable thanks to the combination of efficient algorithms and a complete technological ecosystem.