In today’s digital ecosystem, prices and fares are the most valuable asset of any e-commerce or travel business. However, unfair competition uses increasingly sophisticated bots to extract this data in real time, rotating through datacenter IPs, VPNs, and residential proxies. Stopping them is not a matter of a single filter but of a layered strategy where the first and cheapest layer is IP scoring. In this article we explore how to implement an IP-scoring middleware in Node.js that classifies each request and decides whether to serve, slow, challenge, or block, while being clear about its limitations.
The threat of price scrapers is not new, but their sophistication is. A competitor bot can pull your entire fare table in the time a real customer takes to pick a seat. It hits the same search route every few seconds, reads your prices, and undercuts you on a comparison site an hour later. Blocking it is hard because traffic does not come from a single machine but from thousands of authentic residential IPs that rotate constantly. This article focuses on the first line of defense: scoring every incoming IP before it touches your catalog.
The approach involves building middleware that detects datacenter traffic, VPNs, and residential proxies, assigns a risk score from 0 to 100, and decides the action. The key is not to treat detection as a magic boolean but as a gradient. The same IP can have flags like is_residential_proxy, is_cloud_provider or is_vpn, and the response should be tiered: for example, block for scores above 80, challenge (CAPTCHA) for those above 45, and only monitor for medium ones. This granularity avoids costly false positives, such as blocking a real customer using a corporate VPN.
At Q2BSTUDIO, as a custom software development company, we know that cybersecurity cannot be an afterthought. That is why we design modular solutions where the IP layer is only the beginning. A well-implemented middleware in Node.js, with Redis caching and controlled timeouts, can process millions of requests without affecting user experience. Furthermore, we combine this defense with artificial intelligence to analyze behavioral patterns, and with cloud services on AWS or Azure to scale worry-free. Our team also integrates BI tools like Power BI to monitor traffic in real time and adjust thresholds.
The proposed middleware follows a simple architecture: a scoreIp function queries an external API (e.g., ipgeolocation.io) with a 1.5-second timeout and fails open (if the service is unreachable, traffic is allowed to avoid site paralysis). Then the decideAction function interprets the score and flags to return 'allow', 'monitor', 'challenge', or 'block'. This decision is cached for six hours, avoiding repeated expensive and slow lookups. On a fare site, for instance, an IP from a known commercial proxy (like Zyte Proxy) with a score of 80 receives a direct block on the /api/fares endpoint, while a common VPN (NordVPN) with a score of 60 only gets a challenge at checkout.
But the IP layer is not enough. The most advanced scrapers use residential proxy networks that look like real ISPs. That is where the second layer comes in: rate limiting based on ASN (the network operator), not on individual IP. Since attackers rotate hundreds of IPs within the same ASN, limiting by ASN cuts that movement. In addition, we add behavioral signals like request cadence and honeypots: invisible links for humans that every bot follows. If a bot falls into the trap, its IP is recorded and added to a local blacklist. At Q2BSTUDIO we develop AI agents that learn from these patterns to dynamically adjust thresholds, offering adaptive protection that static solutions cannot provide.
Implementing in Node.js requires correctly handling the client IP behind proxies or CDNs. Setting app.set('trust proxy', 1) and using req.ip prevents scoring our own load balancer’s IP. Redis caching is vital to avoid API cost spikes: a 6-hour TTL is a good balance. For IPv6, you must score on the /64 prefix, not the full address, because a single subscriber can have trillions of addresses. And of course, GDPR compliance matters: IPs are personal data, so set retention policies and purpose.
This layered approach is the same we apply in our cybersecurity projects for clients in travel and retail. We start with the IP layer, which eliminates 80% of malicious traffic (mass scrapers from AWS or Hetzner), then add rate limiting and behavioral analysis. In some cases we integrate Azure cloud services for real-time processing and Power BI for threat visualization. The key is not to rely on a single signal: neither IP blocking, nor user-agent, nor rate limiting works alone. The combination makes the difference.
Finally, remember that detection without logging is a guess. Every block, challenge, or monitor decision should be logged with IP, score, flags, and endpoint. After a week you can adjust thresholds to your real traffic, reducing false positives and improving accuracy. Tools like the AI agents we develop at Q2BSTUDIO can automate that adjustment, learning from patterns of legitimate visitors and attackers. Ultimately, the IP layer is the floor, not the ceiling: a solid, cost-effective first step that any online catalog business should implement.
