When an organization needs to extract value from a massive archive of unstructured documents, the challenge goes far beyond storage. Processing 666,000 pages of PDFs — as happened with the NARA JFK files — and converting them into a fully functional full-text search engine demands a combination of custom software engineering, artificial intelligence, and efficient cloud infrastructure. At Q2BSTUDIO we have tackled similar challenges from a business and technical perspective, designing pipelines that transform opaque data into actionable information.
The starting point is often a mountain of PDFs without search metadata, scattered across folders with no index. For a project of this scale, the first strategic decision is to build a custom application that automates ingestion, validation, and content extraction. A basic script is not enough: you need a fault-tolerant system that handles corrupted files, blank pages, and inconsistent OCR formats. The ideal solution is an orchestrated pipeline in languages like Go, which offers native concurrency and a single deployable binary without heavy dependencies. This drastically reduces operational costs compared to traditional SaaS architectures.
The extraction phase combines several tools: pdftotext for embedded text, Tesseract for OCR when the PDF is a scanned image, and normalization algorithms to clean noise. This is where artificial intelligence can make a difference: natural language processing (NLP) models help identify entities, dates, and semantic relationships within the extracted text. Instead of searching for exact keywords, the user can run complex queries like '(president Kennedy or the president) not (blank page or intentionally left blank) and (top secret or TS/SCI)'. To achieve this, the system must index not only individual tokens but also n-grams and gematric values (Simple, English, Jewish, etc.) that enable advanced searches.
The hosting cost for this type of engine can skyrocket if you opt for managed cloud solutions. A real cost analysis showed that an architecture based on Azure or AWS with services like Elasticsearch, MongoDB Atlas, and managed Redis could exceed $7,000 per month. At Q2BSTUDIO we offer optimized cloud services that dramatically reduce that bill. For example, migrating to an OVH bare metal server for $33 per month, and using a Go-written pipeline to eliminate all intermediate SaaS subscriptions, we achieved savings of $180,000 over three years. The key is designing an architecture that combines local storage with parallel processing using semaphores, avoiding bottlenecks.
Cybersecurity is another fundamental pillar. When handling classified or sensitive documents (like the JFK files), every step of the pipeline must be audited: from secure HTTPS download to secure deletion of temporary files. We implement antivirus scanning with ClamAV on every file before extraction, and detailed logs (debug, info, error) that allow tracing any anomaly. Additionally, using self-generated TLS certificates ensures the final web interface — a standalone search engine served by the Go binary — is only accessible to authorized users.
Artificial intelligence is not limited to OCR or NLP. We can also integrate AI agents that learn from user search behavior to suggest relevant documents or detect hidden patterns in the corpus. For example, an agent could identify that certain cryptic terms systematically appear alongside agent names, helping to uncover non‑obvious connections. This kind of functionality turns a passive search engine into an active investigation tool, especially valuable in fields like OSINT (Open Source Intelligence).
Integration with Business Intelligence (BI) tools like Power BI allows visualizing trends: frequency of mentions per year, density of classified terms per document, heat maps over pages containing more than N cryptonyms. By structuring results in a locally indexed database (rather than a traditional relational database), response times are in milliseconds even with 666,000 pages. The flat‑file architecture with SHA512 checksums and per‑document locks (sync.RWMutex in Go) guarantees consistency in a multi‑threaded environment where hundreds of pages are processed simultaneously.
The complete process is divided into twelve well‑defined stages: from importing the CSV with metadata (identifier, title, URL, local path) to final aggregation of the full document. Each individual page goes through text extraction, PNG image generation in light and dark modes, progressive JPEG conversion, OCR, date analysis with regular expressions, and cryptonym detection. A smart channel (SmartChan) with write control prevents panics in Go and enables smooth communication between the goroutines executing each stage. The result is a structured directory that the reader (another binary) interprets as a self‑contained database.
For companies that need to process huge volumes of documentation — legal, financial, historical, or compliance — this approach is replicable and scalable. It does not require a huge DevOps team or an exorbitant monthly cloud subscription. With an initial investment in custom software development (Go, pipelines, AI‑powered OCR) and a basic server, any organization can build its own full‑text search engine, with the possibility of adding artificial intelligence layers, advanced cybersecurity, and Power BI dashboards. At Q2BSTUDIO we have demonstrated that it is possible to go from $7,000 per month to $33, maintaining — and even improving — functionality. Technology is within reach of those who dare to build, not just buy.
