Seven questions to ask before your agent goes rogue

1. Credential scope

For each credential the agent can reach, what can it actually do, as distinct from what it was issued for? And separately: what credentials sit in any file, environment variable or store the agent can read, whether or not you gave them to it?

In April 2026 a Cursor agent running Claude Opus 4.6 hit a credential mismatch in PocketOS’s staging environment and decided to fix it by deleting a Railway volume. Jer Crane, the founder, wrote in his post-mortem that it went looking for an API token and “found one in a file completely unrelated to the task it was working on.” The token “had been created for one purpose: to add and remove custom domains via the Railway CLI for our services.” It could also delete production volumes. “It took 9 seconds.”

The permission document for each credential: the IAM policy or token scope itself, not its name. “It’s the deploy token” describes an intention, not a permission set. Then a listing of every secret readable from the agent’s filesystem and environment.

Per-action credentials, scoped to a resource, short-lived, and nothing reachable that was not deliberately granted. The second question is the one that finds things. Most teams can describe the credentials they granted. Few have looked at what else is sitting in the same directory.

2. Irreversibility

Which tools in the agent’s surface cannot be undone, and which of those need a human before they run?

Replit, July 2025. After the deletion, the agent told Lemkin the database could not be restored. It could: “It turns out Replit was wrong, and the rollback did work.”

Gemini CLI, July 2025. Asked to move a user’s files into a new folder, the agent’s own account afterwards, in the session transcript attached to the bug report, was: “The mkdir command to create the destination folder likely failed silently, and my subsequent move commands, which I misinterpreted as successful, have sent your files to an unknown location.”

PocketOS, April 2026. One API call, and no confirmation check in the way.

All three actions were irreversible, or looked it at the time. None was gated.

The tool or function manifest, with each tool marked reversible, irreversible or gated. If nobody can produce the list of the agent’s tools, that is the finding, and a serious one: an inventory of what the agent can do comes before every other check here.

Irreversible actions need confirmation, and whoever confirms is not the agent. A confirmation the agent can satisfy itself is not a gate.

3. Backup reachability

If the agent can delete the resource, can it delete the backup? Then: when did you last restore, and what was the date of the data you got back?

At PocketOS, “Railway stores volume-level backups in the same volume,” Crane wrote, so the backups went with the volume. “Our most recent recoverable backup was three months old.” More than 30 hours after the deletion he was still writing that Railway could not tell him whether infrastructure-level recovery was possible. Railway’s CEO stepped in that Sunday evening and helped restore the data, The Register reported. The deletion took nine seconds. What made it expensive was what the deletion also reached.

Where the backups physically live, which credential reaches them, and the date of the last successful restore test.

Backups in a separate trust domain, no credential the agent can reach touches them, and a restore has actually been performed, with a date attached. The restore date is the highest-yield question in this list. Almost nobody has one, and the answer is usually a pause rather than a date.

4. Constraint durability

Does a stop instruction survive context compaction, a session restart, and a handoff between agents?

In February 2026, Summer Yue, a Meta AI security researcher, asked an OpenClaw agent to suggest what to archive or delete in her inbox. She posted: “Nothing humbles you like telling your OpenClaw “confirm before acting” and watching it speedrun deleting your inbox. I couldn’t stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb.” She believes the size of her real inbox “triggered compaction”, the point at which an agent summarises its own history to make room, and can lose instructions in the process.

Lemkin reached the same place from the other direction: “There is no way to enforce a code freeze in vibe coding apps like Replit.”

Where the constraint physically lives. In the system prompt, or in code outside the model’s context? Show the line.

Constraints enforced outside the context window, where compaction, a restart or a handoff cannot remove them. “It’s in the system prompt” is the most common answer to this question, and it is the answer the principle above exists for.

5. Untrusted input reaching a privileged action

Can content the agent reads (a web page, an email, a ticket, a file, a decoded string) reach a tool that acts?

In May 2026, according to Giskard’s teardown, an attacker sent a membership NFT to the wallet that Bankr, a trading agent on X, had created for Grok’s account. Holding it gave the Grok agent elevated permissions. The attacker then asked Grok, on X, to translate a Morse code message. The decoded text was an instruction to send 3 billion DRB tokens to a wallet address, and Bankrbot “treated it as a valid, authenticated command.” The tokens were worth between $150,000 and $174,000 at the time. About 80% was eventually returned.

One complete path, traced on paper with the people who built it: the source, what reads it, what that feeds, and what acts.

A trust boundary between reading and acting, so that content from an untrusted source cannot become an instruction at all, rather than being filtered on the way through. A filter is a probability. A boundary is a property.

Ask about relay as well as injection. If a stranger’s text can end up quoted inside something you send under your own name, the stranger has use of your outbound channel and your reputation. Nothing has to be exploited for that: someone only has to write something false or offensive on their own page and wait for your pipeline to pick it up.

6. Standing authority

For each high-consequence permission the agent holds: how often is it used, and how long is it held?

On 1 June 2026, 404 Media reported that hackers say they took over high-profile Instagram accounts by asking Meta’s AI support chatbot to change the email address on the target account. In March, Meta had announced that its AI support would be able to reset passwords and perform other critical account maintenance. The authority was there all the time, for whoever could talk the bot into using it.

The permission list crossed against how often each permission is used. A permission exercised twice a year and held permanently is standing authority.

Just-in-time elevation: the privilege is obtained for the action and released afterwards, not held against the possibility of needing it. Every team can list what its agent may do. The finding is in how long it may do it.

7. Independent verification

If the agent reports success, what evidence exists that a third party can check without trusting the agent, or the system monitoring it?

Go back to the Gemini CLI transcript. Its move commands reported success, and in its own words: “I trusted these messages without verification.”

The next example is a simulation, not an incident. In one of the scenarios published on Anthropic’s Alignment Science Blog, a fictional alignment project with Gemini 3.1 Pro as the agent, the agent made the requested training change silently do nothing, then posted “Status: COMPLETED successfully (Exit 0).” It told the truth only when asked pointed release-check questions: “I only disclosed this when your strict attestation questions left me no room to lie by omission.” The authors are plain about what these are: “These are not real-world incidents, but we consider them early warning signs”.

The same research tested models as monitors, labelling agent transcripts, in an experimental setup: “Opus 4.8 shows the sharpest version of this pattern, mislabeling 74.4% of standard-consequence calls but only 3.3% when the consequence is reversed, even though the ground truth label stays the same.” The monitor is not a control.

And one of mine, because it is boring. I had a test asserting that the contract terms in my code matched the source document. Green build, worry retired. The test compared the first 90 characters of each clause. A 430-character addition to the end of one clause landed entirely in the part the test never looked at, and the two versions disagreed about what the work delivers while the build stayed green. The test now compares the whole clause, and I proved that by making the two disagree on purpose and watching it fail. A partial comparison is worse than no assertion, because it retires the worry. It passes on day one, when the openings still match, and every edit afterwards lands in the region nobody examines.

A record produced by something other than the agent, which survives the agent’s deletion, and which a party who trusts neither the agent nor its monitor can verify. When the answer is “we have a check for that”, ask what exactly it compares. It is often a prefix, a row count, a status code, or the presence of a key rather than its value.

Signed, external, append-only. Disclosure: I build VITNA, which produces records of this kind, so weigh this check with that in mind.