Breaking the scale
barrier
How we helped Consilio bypass the Elasticsearch performance cliff — by refusing to make it write faster.
Stuck between a wall and a ceiling
In e-discovery, legal teams search, filter and tag millions of documents with absolute precision. For Consilio the hard part wasn't finding the right documents — it was permanently saving which ones got tagged Responsive, Non-Responsive or Privileged without grinding the system to a halt.
Their original setup ran update-by-query directly in Elasticsearch: over an hour to tag 500,000 documents. The workaround — storing tagged IDs in SQL and injecting them back into queries — held up for small batches, then ballooned past 100MB payloads and crashed once files scaled into the millions. ES|QL slammed into a hard 10,000-result cap with no pagination.
A slow write-performance wall on one side. A massive query payload ceiling on the other.
The great decoupling
We didn't try to force Elasticsearch to write faster. We changed the rules of the game — letting each technology do only the thing it is actually good at.
Elasticsearch reads
One single, highly optimized index, used strictly for fast read-only searching. Nothing writes to it during a session, so nothing can slow it down.
Read-only indexSQL remembers
A short-lived session store. The search runs once, the resulting document IDs land in SQL immediately, and the working set lives there for as long as the session does.
Session storeEverything else follows
Pagination, sorting, filtering and tagging all happen inside SQL. No heavy write back-and-forth, no bloated query payloads, and zero performance degradation at scale.
No writes at search timeBefore we teamed up, Consilio had burned real engineering cycles on ID-injection queries and ES|QL joins. Even pushing a split-index metadata approach to its absolute limit together, we capped out at 1,698 updates per second — a massive improvement, still not enough for their real-time volumes. Rather than chase minor gains on a write-heavy design, we pivoted. Recognizing that Elasticsearch is built for reading, not rapid massive writing, was the key that unlocked the SQL-session architecture. Along the way we also killed a subtle tokenizer bug — special characters like a Unicode ellipsis in a client's last name weren't treated as whitespace, quietly vanishing valid results. Patched, and accuracy restored across the board.
From fighting limits to not having any
We treat every engagement as a partnership. What started as a narrow project on document tagging persistence quickly expanded into a full, top-to-bottom cluster architecture review of their Kubernetes infrastructure.
We also went beyond the original Statement of Work to run deep-dive diagnostic reviews on two of Consilio's other major products — Sightline and its successor AVR — making sure the entire product suite was set up for long-term stability.
"…And Justice for All" — Metallica. Dense, complex, a marathon to get through. The right backing track for untangling millions of legal documents.