Where the data may go picks the model

A client who runs leadership assessments asked me this week whether AI could condense a thirty-page 360-degree feedback report into half a page: three strengths, three areas for development, a few recommendations. A consultant would edit every summary, and nothing written by a model would ever go to a customer unreviewed. The reason for asking was price. They had recently lost a tender to a competitor who charged half as much. The condition came in the same breath: the data must never leak, and must never be findable on any public AI platform.

Never findable, translated

The condition is reasonable and, as stated, cannot be checked. Nobody can verify "never findable". What can be checked is a short list of questions about where the report goes once it leaves the consultant's laptop.

• Is the content used to train anything?
• How long is it stored, and where?
• Who can read it along the way: the provider's staff, their subcontractors, anyone with access to logs?
• On which machine is the computation actually done?

A hosted assistant with a proper data processing agreement answers the first three on paper, and the answers are usually acceptable. The fourth answer stays "on someone else's machine", and for a report that names a manager and quotes what their team thinks of them, that is the one the client was worried about. A model that runs on a machine the firm controls answers all four with "here".

It also makes the firm the operator. Somebody patches that machine, decides who can log in, keeps the backups, and knows which laptop the reports were copied to. That responsibility is the real price of the local option, and it belongs in the recommendation next to the licence costs.

Half a page has to be checkable

The summary only saves money if editing a draft takes less time than writing one from scratch. That depends almost entirely on how quickly a consultant can check what the model wrote.

This week offered a useful reference point. A major provider launched a legal research product with a dedicated index of over 200 million legal documents. With that index, the share of correct answers rose from 38.7 to 54.0 percent. That is real progress, and it still leaves almost half the answers wrong in a domain with far better source material than a single feedback report. For an assessment, a wrong item under "areas for development" is a false statement about a named person, delivered by the consultancy.

So the review is part of the product, and it should be designed as carefully as the prompt. My working version: every statement in the summary points to the pages of the source report it is based on. The consultant then checks three strengths and three weaknesses against six page references, which takes minutes. A summary without references has to be checked against thirty pages, and at that point the time saving is gone.

Small models for narrow jobs

Until recently, "run it locally" meant accepting a much weaker model. That gap is closing for narrow tasks. Another release this week claims that a 27-billion-parameter model compressed to under six gigabytes keeps about 98 percent of its full-size benchmark performance, small enough for a well-equipped laptop. That is the vendor's number and it was measured on reasoning benchmarks, which say little about German feedback reports.

A structured summary with a fixed format is about as narrow as a language task gets. Whether a local model is good enough for this one is a question one real report can answer, and a leaderboard cannot.

What I will test before recommending anything

• One real report, anonymised, run through a local model with the page-reference format. The same report summarised by a consultant without AI, timed.
• The consultant then edits the model's draft, also timed. The comparison that matters is minutes of consultant time per report, before and after.
• A written list of where the report travels in the local setup: which machine, which folder, who has access, what gets deleted and when.
• For comparison, the same four questions answered for one hosted option under a data processing agreement, so the client decides between two described paths and not between a fear and a promise.