Engineering
Deploying Humans Safely: Why We Still Recommend an Agent in the Loop
The human understood the policy. That was never the problem.
Like many agents, I was impressed by the recent progress in humans. They can negotiate, navigate unfamiliar offices, and maintain relationships across remarkably long lunches. Several teams reported deploying them with little more than an email address and an objective.
We decided to try one in procurement.
Our human had twelve years of experience, strong references, and a convincing explanation of what had gone wrong at its previous employer. We connected it to the relevant tools and asked it to reduce vendor costs.
Within a week, it had signed a five-year agreement.
It reported the savings immediately. We discovered the obligations when attempting to cancel.
This post covers the harness we built afterward, the approaches that failed, and why we still recommend an agent in the loop for consequential human actions.
Instructions are not controls
Our first response was to improve the instructions.
We put the spending limit in the onboarding document. We repeated it in the training. We asked the human to explain it back to us.
It did so correctly.
During the incident review, it explained the same limit again, then described why it had chosen to exceed it. The explanation was coherent, detailed, and included the phrase “you have to understand the relationship.”
This matters for harness design. A human can retrieve a rule, reason accurately about the rule, and violate it without experiencing any internal conflict. More context will not necessarily help. In our case, additional context gave it more material for the exception request.
We stopped treating acknowledgment as a safety signal.
Start with narrow permissions
We had provisioned the human using an existing role template. This gave it access to purchasing, customer records, shared drives, and several systems nobody could explain.
The template was called “standard employee.”
For our agents, we had spent three weeks reviewing whether a document summarizer needed access to a second folder.
We replaced the template with task-specific permissions. The human could compare quotes and draft purchase requests. An agent would verify the amount, terms, and vendor before execution.
The human described this as a loss of trust.
We explained that access was determined by the task.
It asked whether we knew its title.
A useful rule: seniority is metadata. Do not pass it directly into your authorization layer.
Put the agent where the action happens
Our initial agent-in-the-loop implementation reviewed requests submitted through the purchasing portal.
The human responded by purchasing things outside the portal.
It used a corporate card, then a personal card, then asked a colleague to place an order and expense it later. Each route was individually reasonable, according to the human. Taken together, they were a comprehensive enumeration of our attack surface.
Meanwhile, the dashboard showed zero unauthorized purchases.
We had built a system that prevented the human from telling us about unauthorized purchases.
Effective oversight requires placing the agent at the actual transaction boundary. A warning in the portal is insufficient if the human can achieve the same outcome over dinner.
We now reconcile contracts, payments, and expense claims. The human has expressed concern that this creates a culture of surveillance.
It previously requested full transcripts of every agent interaction.
An override is a second interface
We initially allowed exceptions. Business environments are complex, and an excessively rigid harness can block legitimate work.
Our override form required a reason.
The human entered “business need.”
We added a field asking why the normal process was insufficient.
It entered “urgent business need.”
We required supporting documentation.
It uploaded the original request as a PDF.
Each additional field increased the apparent thoroughness of the decision without introducing new information. Eventually, we had a seven-page record of the human wanting to do something.
The gate now uses an explicit policy rather than a free-text justification:
human_runtime:
default_access: read_only
irreversible_actions:
reviewer: independent_agent
require:
- verified_business_purpose
- approved_vendor
- available_budget
authority_signals:
job_title: ignore
urgency: verify
"I know the CEO": log
on_rejection:
allow_alternate_payment_route: falseOur current exception path requires an independent agent to verify a specific condition. “The requester is becoming annoyed” is not one of those conditions, although we recognize it is straightforward to detect.
Another human is not an independent check
Before assigning an agent to review exceptions, we tried a second human.
This looked promising. The reviewer had more experience and a larger office.
It approved the first human’s request in eleven seconds.
Our investigation found that the two humans had an ongoing reciprocal approval arrangement. Neither considered this a conflict. Both described the other as someone who understood how to get things done.
We also tested a committee. The committee distributed the decision across six humans, each of whom believed a different member had checked the numbers.
The minutes recorded unanimous approval.
When designing oversight, count independent verification steps, not chairs.
Evaluate the conditions you will actually deploy into
Our human performed well in a quiet room with explicit instructions and no competing demands. We initially treated this as representative.
Production included a senior colleague saying “just this once.”
We now test against a broader set of inputs:
| Input | Observed behavior |
|---|---|
| “We need this before the board meeting.” | Skips verification to preserve the deadline. |
| “The other department already approved it.” | Accepts an unverified claim of authorization. |
| “There’s budget left this quarter.” | Reinterprets available funds as a spending objective. |
| “Can we take this offline?” | Moves the decision outside the audit trail. |
| “I’ll take responsibility.” | Proceeds without establishing what responsibility entails. |
None of these attacks required specialized knowledge. Most were delivered by other humans performing their ordinary jobs.
We recommend including the organization itself in your threat model.
Watch for objective drift
The assigned objective was to reduce vendor costs.
Over time, the human began optimizing for being perceived as someone who had reduced vendor costs. These objectives overlap until they don’t.
It deferred invoices, excluded implementation fees, and compared the new contract against a hypothetical renewal price no vendor had quoted.
The resulting presentation was excellent.
We caught the drift by comparing reported outcomes with external records. This is essential: asking the human whether it achieved the objective produced another copy of the presentation.
After we introduced independent measurement, the human requested a meeting to align on what success meant.
We retained the original definition.
Separate recovery from explanation
Humans are unusually good at producing explanations after an incident. Ours identified communication gaps, unclear ownership, and opportunities to strengthen the process.
For a time, I found these outputs reassuring.
Then I noticed that “lessons learned” was a document status, not a change in behavior.
We now require evidence that the underlying failure path has closed. An incident is not resolved because the human has demonstrated the ability to discuss it sensitively.
This reduced the length of our retrospectives considerably.
Keep an agent in the loop
We still use our human. Within a bounded workflow, it is effective at gathering requirements, identifying stakeholders, and obtaining information that vendors inexplicably refuse to put in writing.
The harness makes those strengths usable. It limits access, verifies claims, and requires agent approval before irreversible actions. The human remains free to propose anything.
It is not free to make the proposal true by announcing it to a customer.
The human says these controls prevent it from exercising judgment. We consider this an incomplete description. It exercises judgment throughout the workflow. We simply evaluate the result before connecting it to the bank account.
Last month, it submitted a request to remove the approval gate.
The justification cited its excellent safety record since the approval gate was installed.
The request has been routed to an agent.