Search finds words. People remember facts. The gap between those two is where a message archive quietly fails its owner: you know the roof got discussed, you know a number was agreed, and none of the words you can think of appear in the message that matters. For an app whose whole pitch is "your record, kept," the worst failure mode isn't a wrong answer. It's "I don't see anything about that" when the answer is sitting right there in the archive. We came to call this the false denial, and most of the engineering below exists to make it rare and, when it happens anyway, visible.
The citations contract
We started from the answer's obligation, not the model's capability: an answer over your own history must be checkable in one click. The mechanism is deliberately boring. The transcript we send is numbered, one line per message. The model is instructed to end its reply with a line like CITATIONS: [12, 47] naming the lines it relied on. The app strips that line, maps the numbers back to message ids, and renders each citation as a row you can click to jump to the real message in the real thread, exactly the way a search hit jumps.
The important property: the map from numbers to messages never leaves the machine. The model sees line numbers; it cannot see, invent, or leak ids. If it cites a number that doesn't exist, the mapping fails app-side and the phantom citation is dropped. If it repeats itself, we dedupe before the eight-citation cap. The model composes; it does not get to notarize.
The UI half of the contract took a design round of its own. Eight citation rows pushed the answer itself off screen, so citations fold: a closed-by-default row headed by what the answer stood on, like "Based on 5 messages and 1 document from 3 conversations." An early cut showed just a count and got rejected in review as a stray number. The row has to say these are the evidence, or citations stop reading as the thing that makes the answer trustworthy.
The pipeline we built, shipped, and deleted
Semantic retrieval needs embeddings, and embeddings need a model. The first architecture was the obvious one: the app batches message text through our relay to Google's embedding endpoint, and the vectors come back to live in a local index. We built the whole thing properly: a Worker endpoint with daily rate bounds, a lifetime allowance with credit metering past it, kill switches, quota tables. It worked.
Then we deleted all of it, and the reason is worth recording. Small embedding models crossed a line: EmbeddingGemma, running locally via ONNX Runtime, embeds around 150 messages a second on an ordinary machine (a 100,000-message archive indexes over a coffee), and on our 400-query retrieval benchmark its quality was indistinguishable from the cloud embedding model on the same data. Getting there was its own saga of ONNX graph surgery and quantization trade-offs; the short version is that the build of the model file mattered 3 to 4x more than anything else we touched. Once that was true, every remaining argument for the server was our convenience, not the user's. The moment the local path shipped, the cloud path stopped being a fallback and became a liability: a second code path to test, a second privacy disclosure to maintain, a second thing that could silently behave differently. So the relay's embedding endpoint isn't dark or deprecated. It's gone. The index is built here, always.
What that buys is a privacy story you can state in one sentence: building the index sends nothing anywhere. You can index an archive with networking off. Only asking a question transmits, and what it transmits is bounded and visible: the question plus the retrieved excerpts, through the same hold-nothing relay summaries taught us to run, to a provider under zero-retention terms. The consent dialog says exactly that, once per archive, and the off switch deletes the index.
Retrieval tricks that earn their keep
A question is embedded on-device and cosine-scanned against the index. The refinements all trace back to a specific observed failure:
- Every question is embedded twice: once alone, and once with the previous question appended. Both get searched and the retrieval slots are split between them. A follow-up like "and what did she say about the deposit?" keeps its anchor in the conversation that "she" refers to, while a hard change of subject can't be outvoted by the topic before it. This one fix closed a whole class of false denials.
- A literal hit must not be outbid by a mediocre cosine one. If your question contains words that literally appear in a message, that message earns its slot. Embeddings are for what words can't reach, not a replacement for the search that already worked; the two lists share the budget rather than compete on one score.
- Hits bring their neighbourhood. A retrieved message arrives with a window of surrounding messages in chronological order, because "yes" is not evidence of anything without the question above it.
- Archive-wide asks group before they spend. The top hits across the whole index are grouped by conversation, capped at eight conversations, each contributing a windowed section. The answer can draw on a text from 2021, a group chat, and last month's PDF at once, and each citation carries the conversation it came from.
A contract is not a message
Attachments broke our tidy data model. A message is one retrievable thing with a sender, a date, and neighbours. A 40-page disclosure PDF is dozens of retrievable things with none of those. The fix has three parts. Document chunks get their own id namespace (message ids live low, WhatsApp's ride at 240, document chunks start at 248, with page and chunk packed into the low bits), so a retrieved id still announces what it is. Extraction runs per page: the PDF's own text layer where one exists, text recognition for pages that are just pictures, and for image attachments, which have no text layer to try. And because a chunk id has nothing to hydrate through the message store, extracted text lives in a sidecar file beside the index, framed per document with the page it came from and whether it was read or recognised, a distinction the citation row shows the user, because OCR of a phone photo of a countertop contract deserves less trust than a born-digital PDF.
Two details we'd defend in any review. The sidecar is encrypted under the same per-backup key as the index rows: this is document text out of somebody's legal or medical life, and storing it in the clear beside an encrypted backup would leak exactly what the password protects. And coverage is committed index-rows-first, sidecar-second: a crash between the two leaves rows that fail to hydrate and get re-read next run, where the opposite order would mark a contract as covered while leaving it silently unsearchable forever. That's the false denial again, hiding in a commit ordering.
Honesty is load-bearing
The same rule that governs exact search governs Ask: the feature may be less capable than we'd like, but it may not lie about what it did. If the index lags the archive, the answer says what date it's indexed through. If there's no index at all, the answer says it only saw recent messages instead of silently pretending it read everything. If the AI service is unreachable, Ask says so and exact search keeps working, because it never depended on anyone's uptime. And a standing line under the pane says answers can be wrong and points at the citations. None of this is legal boilerplate; each label traces to a session where we watched a plausible answer earn more trust than it deserved.
The arithmetic under the hood
One question costs one AI credit, same as a summary, but they are not the same workload: a summary chunk approaches 240,000 characters of input where an ask sends a few thousand. That difference is why the relay routes asks and summaries to different models server-side; the headroom on an ask pays for a stronger model where it actually improves answers, without touching the app. The relay itself is unchanged from the summaries design: device-token auth, streaming, input caps, credit accounting server-side, message content never stored or logged, error classes only. A model swap remains a config change, not a release.
The feature this all adds up to looks small: a pane, a question box, some rows under an answer. That's the point. The size of the answer's claim is capped by the receipts under it, the size of the send is capped by retrieval, and the index that makes it work never leaves the machine it describes. The user-facing story is in the announcement post; the disclosure lives in the privacy policy.