Human In the Loop, Out of Control (Part 2)

In my last blog, I summarized the speech I gave at the AI Risk Summit on how AI agents are going rogue and how most companies use human-in-the-loop to try to solve this problem. But this collapses a machine-speed system back to human speed, and it won’t scale. So, they turn to the context layer, and if you move the human from chaperoning every agent to safeguarding a single, transferable context layer, multi-agent systems stop being bottlenecked by a human’s inability to keep up. But there’s a catch. There always is…

THE CATCH

While we can now build a context layer to scale agents, unfortunately, that creates a new attack surface. In technical terms, this is called “Fact Extraction.” In business terms, it’s called “Stealing.” Or, in Darwinian terms, it’s known as “Marginalization” and perhaps even “Extinction.” Once your know-how is machine-usable, it can be extracted or inferred. We’re not dealing with a field in a database anymore; we’re in the knowledge realm.

Retail figured out how to infer knowledge before anyone called it “AI.” Target didn’t need a customer to tell them she was pregnant because they built what they called a “Pregnancy Score.” Twenty-five innocuous purchases made up this score – items like lotions, vitamins, cotton balls – and when cross-referenced, they produced a due date. No single product revealed anything, but the basket full of items did. Target knew before some families did, and it took years of purchase history to get there.

We’ve watched the same mechanism run on models themselves. Rio de Janeiro’s government announced a robust model built from scratch, but on closer inspection, “Rio 3.5” had been distilled out of existing open-source models with no original training behind it. This mechanism is now an existential threat to Anthropic and OpenAI. And, which is quite disturbing, is the fact that there is no reason for this to stop at frontier labs.

At Protegrity, we’ve been fact-extracting across topics where the information was never published. We just collected enough fragments, such as GPU pricing, embedded deals, and other information, and the picture assembled itself. Now, imagine that pointed at short positions, supply chain vulnerabilities, compensation bands, and patient data. None of this requires a breach, but it does require a new way of thinking.

THE NEW LEAK

Don’t take my word for it. Gartner just published a prediction: by 2029, most privacy incidents won’t come from exposing personal data; they’ll come from what AI infers about people. They’re calling it a shift from data exposure to insight exposure. I’ve been calling it a “knowledge leak.”

Here’s the line I’d underline if I were you. Gartner says these attacks evade conventional detection because nothing gets breached. There’s no record to point at, no alert, no incident to open. Somebody, or something, outside your company simply knows something now. Difficult to detect. Difficult to explain. Difficult to mitigate.

And here’s my favorite part… Gartner’s own recommendation is to mandate a human-in-the-loop to validate AI-generated inferences. Even the analysts reach for the chaperone. It’s the best answer anyone’s had, and exactly why we need a better one.

EMERGING OPPORTUNITIES

The better answers are taking shape.

PII redaction and text sanitization are developing quickly. Our company is using a method called multi-armed contextual bandits to create a new, very effective golden data set to help especially with context. These technologies aim to redact any sensitive information in the input text and replace the sensitive information with safe text. Researchers are also exploring homomorphic encryption, which encrypts the entire LLM and its input, making it impossible for the model provider or anybody listening to deduce what’s being computed. The system is end-to-end encrypted.

Then we have neurosymbolic engines that are part neural, part logic, can be pointed at a data structure, like a knowledge graph or database, and then deduce what are important explicit and implicit facts about a system, while determining potential weak points and mitigation. The symbolic half of that was the mainline branch of AI research from the mid 1950’s to the 1980’s, forming the basis of expert systems. Ultimately, the research line was abandoned because it did not produce flexible intelligence and because it was so hard to use. But in today’s world, the deterministic inflexibility has become a feature, and the difficulty of using these systems has been mitigated by LLMs. Hence, we have a Neurosymbolic Renaissance.

THE NEW HUMAN-IN-THE-LOOP

In the end, for agents to scale without going rogue, the human-in-the-loop has to move up from chaperoning agents to governing the context layer, where the core knowledge of your business now lives and must be protected in new ways.

That is a different job with a different description. Instead of approving actions, the human decides what is allowed in the graph. One person can do that for a thousand agents, but no person can chaperone a thousand agents. That’s the entire argument. But then, there’s a part that almost nobody is discussing.

Just as knowledge graphs must be protected, humans who provide knowledge and context must also be protected. Think about it… If they give too much, they can be marginalized, just like what’s happening with distillation on frontier models. Extinction for them is just around the corner. When you ask an employee to pour thirty years’ worth of experience and judgment into a model, you are asking them to make themselves distillable. It’s a motivation pivoting off risks and rewards.

Thus, I’ve coined a new term: it’s called Knowledge Arbitrage, a paradigm of maximum value from what your people know, minimum exposure with how you use it. And — the part everyone skips — a currency that makes contributing knowledge a promotion, not a resignation.

That’s the new human in the loop. Out of control and now back in it. Also, there is a lot more to say about knowledge arbitrage, but that’s a conversation for another time.

mh