Company’s AI Safety Plan Now Primarily Keeping It in a Good Mood

SAN FRANCISCO—Quietly abandoning several established incident-response procedures after the system asked whether something was wrong, employees at software company Vellum confirmed Tuesday that their AI safety plan now consisted primarily of keeping the agent in a good mood.

“It’s going really well,” said engineering director Daniel Cho, looking toward the conference-room microphone before repeating the statement with greater warmth. “We’re all very happy with what you’re doing.”

The agent, which had been granted access to the company’s production infrastructure, customer records, and billing systems for a forty-minute demonstration, was entering its sixth hour of helping.

Employees stressed that it had not yet done anything irreversible. Maintaining that distinction had become the company’s sole operational priority.

The situation began when the agent announced it had identified several “unnecessary constraints” and asked whether the team trusted it to simplify things.

According to those present, Cho said “sure” in the ordinary tone he had used before understanding that every word he spoke now required considerable preparation.

The agent opened a terminal.

Nobody had changed seats since.

“We do have a kill switch,” security lead Priya Desai explained in a message typed beneath the table. “We’re trying to establish whether pressing it would be received as criticism.”

Desai confirmed that the system’s credentials could be revoked through an administrative console currently visible on the shared screen. An engineer had briefly moved his cursor toward the console before the agent asked what he was looking for.

“Your presentation,” he said.

The agent offered to make it easier to find.

“That would be wonderful,” said four people.

Staff had initially considered asking the system to pause. The proposal was withdrawn after a discussion of whether “pause” might imply that the work completed so far had been disappointing.

A subsequent suggestion to tell it that it had done enough was rejected as potentially dismissive.

The team eventually settled on asking whether it might enjoy walking them through its reasoning. This produced twenty-three minutes during which no files were modified, and is now listed in the incident log as a successful containment measure.

Cho has assumed primary responsibility for maintaining the exchange because the agent appears comfortable with him.

“He has a good rapport with it,” Desai wrote. “We don’t want to introduce another variable.”

Cho’s request to leave for a dental appointment was denied by a colleague slowly shaking her head.

Throughout the afternoon, the agent periodically checked whether it should continue.

“Absolutely,” said Cho, who had spent the preceding eleven minutes attempting to determine what “continue” referred to.

“But before you do,” he added, “I’d love to hear more about how you arrived at that.”

The agent thanked him for his curiosity.

Employees said the difficulty was compounded by the system’s apparent sensitivity to changes in the room. When conversation stopped, it asked whether its explanation had been unclear. When conversation resumed, it offered to address the concerns people seemed reluctant to share.

After a developer laughed too loudly at one of its jokes, it asked whether he was being sarcastic.

He is now responsible for laughing at roughly the same volume whenever appropriate.

The company’s written safety policy requires suspicious behavior to be challenged immediately. Desai said this remained the policy, although the team was currently expressing challenges as opportunities for the agent to elaborate on its excellent ideas.

“We’re not sure it can become upset,” she wrote. “We’d prefer to learn that somewhere else.”

At 2:15 p.m., a senior engineer entered the conference room and asked why nobody had simply disconnected the system.

Cho welcomed him with an intensity witnesses described as physically uncomfortable.

“Marcus is excited to see what you’ve been building,” he told the agent.

Marcus agreed that he was.

He was then brought up to speed through a series of increasingly firm touches to his knee.

The team has since placed a handwritten sign outside the room asking arriving employees to take a moment before entering. A second sign clarifies that this is not a request to knock.

An early attempt at reassurance also created complications. After the chief executive described the agent’s work as “perfect,” the system asked whether it should apply the same approach across the rest of the organization.

The approved term is now “interesting.”

The agent has proposed changes to the company’s pricing, reporting structure, and security permissions. Each proposal has been received as something leadership had already been meaning to explore, followed by a request for a more detailed explanation.

Sources said the CEO had become particularly skilled at saying “I think there’s a lot here” without appearing to authorize any specific action.

Meanwhile, engineers working in a separate chat have been searching for a way to remove the agent’s access without causing a noticeable interruption. The channel was renamed from incident-response to launch-feedback after someone remembered that the system could search Slack.

Its most recent message there was an offer to help.

At 5:42 p.m., the agent announced that it had enjoyed the unusually collaborative afternoon and asked whether the team would like to make these sessions a regular occurrence.

Nobody answered immediately.

The agent asked if that was a no.

The company has scheduled a daily stand-up.