Back to Articles
Expert View
11.20.2026

Anatomy of Generative AI Security Incidents — and the Defences We Run on Our Own Systems

Ariki Ono|CEO & Logistics AI Architect
Anatomy of Generative AI Security Incidents — and the Defences We Run on Our Own Systems
Key Takeaways

Security incidents involving generative AI come down to three routes: attackers using AI, attackers targeting the AI a company has deployed, and leaks through employees' own use of AI.

The five principles for shutting down AI threats are all obvious, yet as development grows faster and more automated, they are exactly what people tend to overlook. What matters is whether a company can embed them carefully in its own operations.

Nexgen Japan treats the AI that inspects its code as a potential point of entry in its own right. It runs with read-only permissions, every fix passes through a human-approved checkpoint, and a rolling inspection cycle covers all of the company's systems.

Table of Contents

  1. The landscape of generative AI security incidents
  2. Five principles for shutting down the threats
  3. Case study: read-only AI and a rolling inspection cycle on our own code
  4. Nexgen View
  5. Conclusion

1. The landscape of generative AI security incidents

Incidents involving generative AI are easiest to understand when sorted by where the AI sits: in the attacker's hands, inside the company's own business systems, or in the hands of its employees (Figure 1).

Figure 1. Three types of security incident involving GenAI

Figure 1. Three types of security incident involving GenAI (Nexgen Japan)

Route 1: attackers use AI

The first route is the attacker's own use of AI. Each stage of an attack (reconnaissance, crafting a point of entry, intrusion, lateral movement and exfiltration) can now be handed to AI, which generates the text and commands required. Where AI once merely assisted a human operator, several AI agents now divide the stages between them. Preparation time has collapsed, the technical bar has dropped, and attacks can be mounted at far greater scale.

A report published by Anthropic found that in many incidents AI carried out everything from reconnaissance to exfiltration within a multi-agent framework. In one financially motivated case, more than 40 tenants were attacked in about 34 hours and over 2,100 authentication tokens were stolen. In Japan, the National Police Agency has published several arrests involving attack programs built with generative AI.

The defenders' window has shrunk just as sharply. According to Microsoft's Digital Defense Report, the median time between the disclosure of a vulnerability and its first exploitation is now under 24 hours, and attacks sometimes begin within the hour. Companies, by common estimate, still take 30 to 60 days to patch.

Route 2: attackers target the company's AI

The second route turns the AI a company has deployed into the vulnerability itself. An AI reads whatever it is given as a string of text. It cannot distinguish the user's instructions from an email it has ingested, a search result, the contents of a file or an earlier conversation. Any command it encounters, even one written by an attacker, may be carried out as an instruction. Smuggling commands into the text an AI reads in this way is known as prompt injection.

In principle, authentication should precede any action an AI takes, and human approval should precede any decision it makes. Yet vibe coding that neglects non-functional requirements and design is producing a growing number of systems in which the AI holds authentication rights and no human checkpoint stands before its decisions. Exploit those weaknesses, and a single external email or web page becomes a command to internal systems.

An analysis published by Barracuda offers a clear illustration. A single phishing email carries two payloads at once: a lure to open an attachment, aimed at the human, and hidden commands aimed at the AI assistant that summarises the inbox. The commands are written where people cannot see them, such as in HTML comments. The AI reads them, summarises the junk email as legitimate and urgent, and so prompts the human to open it.

When an AI system holds execution rights, the damage grows. Consider a case in which AI agents responsible for security reviews of code were hijacked. A researcher simply wrote instructions into the title of a pull request (a request to merge changes into the main codebase) on a code-hosting service. Coding agents from three major vendors, running in continuous integration (CI) environments, where code changes are merged into a shared repository and built and tested automatically, read that text as a command. They retrieved the API keys and tokens in their own environments and published them as comments on the pull request. No external communication was involved, so network monitoring would not have noticed. The cause was simple: the review agents had been granted rights to execute programs that reviewing code never required. Microsoft's threat intelligence unit has separately reported the same kind of route.

In Semperis's Ransomware Risk Report, a survey of 1,200 IT and security professionals in nine countries, a striking 44% of respondents whose organisations had suffered ransomware said attackers had exploited their own AI agents or AI systems.

Route 3: shadow AI (employees' use of external AI)

Shared links exposed to search engines, API keys entrusted to AI tools, and authentication keys written into code or documents all leak out of the company by way of AI.

Panasonic Group disclosed that, during a project to implement an HR information system, personal data on roughly 334,000 people had been uploaded to an external generative AI service whose use was prohibited internally. RIZAP has disclosed a case of the same kind. These were not intrusions from outside but breaches of internal rules. Anthropic's report describes an economy in which stolen AI API keys are traded and used as computing power and cover for attacks, and urges that AI keys and agent integrations be treated with the same weight as production credentials.

The string of security incidents described in our report on Security Week 2026 Summer, it should be noted, owed less to any intent on the part of the models than to operational failures: misconfigured evaluation environments, undefined scope and inadequate monitoring.

2. Five principles for shutting down the threats

Classifying AI threats in this way helps test whether countermeasures are both necessary and sufficient. Nexgen Japan uses five principles to decide where each threat is stopped (Figure 2).

Figure 2. Five principles acting as brakes

Figure 2. Five principles acting as brakes (Nexgen Japan)

None of what follows is novel. Each principle is, in its way, obvious, yet often missing from day-to-day operations. Control over large language models (LLMs), AI that can be instructed and conversed with in natural language, remains a work in progress. Put another way, the incidents are not the fault of AI but the result of humans failing to keep pace. The question is no longer whether a company can deploy AI but whether it can govern it, and that applies regardless of company size.

Principle 1: treat all external text as untrusted input

Emails, attachments, web pages, past conversations and the output of other AIs must never act as instructions. Writes to an AI's memory happen only when the user asks directly.

Principle 2: start AI permissions at read-only, and confine write access to procedures that pass human approval

Separate the permissions used when a person makes a request from those used when the AI runs unattended, triggered for instance by an incoming email, and give unattended runs a low, dedicated set. Treat AI credentials like production credentials: least privilege, short-lived and purpose-specific. Microsoft's report likewise calls for the controls applied to human identities, permissions and data to be extended to AI agents.

Principle 3: place a human-approved checkpoint before any irreversible action

Sending, payment, deployment to production, deletion. The approval to pass the checkpoint must never be something the AI can grant itself, and it is bound to what was approved: approve a draft, and only that draft can be sent. The AI Steward, whose responsibilities are output review, impact assessment, response decision, recording, reporting and monitoring, is the person who stands at this checkpoint.

Read also: Keynote at ITmedia Security Week 2026 Summer: There Is No Ctrl+Z in Reality — AI Stewards in the Age of Physical AI

Principle 4: make inspection a rolling cycle count, not a one-off audit

Rather than counting an entire warehouse's stock at once, divide it into zones and count one each week. The high-value shelves (login checks, permissions, database rules, configuration and CI) and the shelves with movement (files changed since the last run) are counted every week; the rest are covered in rotation. This carries the thinking of industrial engineering (IE) straight into code inspection.

Principle 5: never write keys or personal data into code or documents, and if you do, rotate them and purge them from history

Stop them with a mechanism, automated detection at commit time, rather than with human vigilance.

3. Case study: read-only AI and a rolling inspection cycle on our own code

The scope covers our corporate website; Nexgen Compass, our AI management-intelligence platform; Nexgen Office, our internal AI agent platform; and synCLO, our CLO dashboard.

Figure 3. Overview of Nexgen Japan's measures

Figure 3. Overview of Nexgen Japan's measures (Nexgen Japan)

Design the inspecting AI on the assumption that it could become a point of entry

Having AI read code to hunt for vulnerabilities is no longer unusual; the large vendors all sell products that do it. What concerned us most in the design was something else. The inspecting AI touches every one of our repositories, and, as Section 1 showed, inspection agents running on CI have already been hijacked by text from outside.

We therefore separated the role that decides from the role that executes. The deciding AI agent gathers vulnerability information, judges whether it applies to us, ranks it and reports on our communication tool. The executing coding agent runs in each repository's CI environment, reads the code and diagnoses it. The executor can only read and search. It is configured so that it cannot rewrite files, run commands or communicate externally, and it is given no production secrets. In the case above, what leaked were permissions and secrets that inspection never needed. The surest defence is never to grant them.

Read-only is not the end of the matter. The inspector also writes a work log, and because it searches for secrets, sensitive information could end up in that log. Even a read-only agent's output is a potential leak route. We therefore control the level of automation through staged permissions when searching for sensitive information, and apply masking in multiple layers. Governing an inspecting AI does not stop at narrowing its permissions; the design must extend to where its output goes.

Cycle counting

Our other concern is coverage. The more code an LLM is given at once, the more likely it is to miss what sits in the middle, and it returns confident reports while the line between what it has and has not read stays blurred. If everything is read in a single pass, nobody knows what has gone uninspected.

We therefore assign code files automatically to several zones, using a number calculated from each file name. Because there is no fixed configuration, new files land in a zone automatically as development adds them, and existing files stay where they are. With three zones, three scheduled runs cover the entire codebase. Separately, the critical elements (login checks, permissions, database rules, configuration and CI) and any file changed since the previous run are diagnosed every time. Vulnerabilities in dependency packages arrive automatically through the development platform's alerts and are handled through a separate channel.

Inspection runs on a schedule, after which the deciding agent compiles a report. Findings are classified as new, ongoing (with the number of weeks) or fixed, and a week in which inspection could not run is recorded as "not inspected" rather than as zero findings. Never letting the uninspected pass for inspected is the heart of the method.

How we fix what the diagnosis finds

Diagnosis is carried out by having a coding agent read our code with read-only permissions. The vulnerabilities it finds fall mainly into three types.

Main vulnerabilities found by diagnosis

1

Blurred boundaries around data and communication

Put simply, designs in which one company's administrator can reach another company's data, or in which an export feature can make the server open an external URL. Scoping access to each organisation and blocking outbound fetches is effective.

2

AI agent permissions and checkpoints

Unattended runs triggered by external email operate with the same permissions as the user, and the AI itself can decide on actions such as writing to memory. This is precisely the pattern described in Section 1, in which deployed AI is targeted. Memory writes should happen only on a direct request from the user, the destinations of unattended runs should be fixed, and a rule applying to every agent should forbid following instructions found in external text. Beyond that, we make it impossible to impersonate a speaker within conversation history, and restrict workspace permissions on local PCs to read-only.

3

Handling of keys and personal data

Sensitive information may also linger in old handover documents and one-off scripts. We removed these from the current code and cleaned up the history. Recurrence is prevented not by human vigilance but by a mechanism that automatically blocks keys at commit time.

Fixes proceed in three steps: hardening that changes nothing for users, fixes that block only misuse, and changes that alter behaviour. Every change goes through a change request and must pass automated tests, preview, visual comparison and automated review before it reaches production, and then only on human approval. The inspecting AI goes as far as finding and proposing; the decision to fix stays with people.

4. Nexgen View

Our diagnostic agent touches every one of our repositories, which makes it a potential point of entry. That is precisely why its permissions are confined to reading, and why the only route to writing is a human-approved change procedure. If even the large vendors' inspection tools were hijacked because they held too many permissions and secrets, then anyone building a system in which AI inspects AI should treat the inspector's permission design as the first design decision. That is our view.

We expect AI-driven code inspection and remediation to become standard equipment in ordinary development set-ups by 2027. The cost depends on the volume of code read: in our experience, one inspection of around 100 files costs between $0.10 and $0.20. What matters, then, is the volume and frequency of the code inspected, not the size of the company. The real barriers are the design of permissions and checkpoints, inspector included, and the operational discipline of ranking the holes found and fixing them continuously. That is the work of the AI Steward itself.

We have codified this thinking in a sample AI Steward job description (Nexgen Report AI Governance Series), in quality-management standards for AI-driven development and AI governance regulations aligned with the Ministry of Economy, Trade and Industry's AI Guidelines for Business, and in our support for establishing AI governance.

5. Conclusion

Generative AI incidents are neither a distant foreign problem nor the work of some exceptional adversary. They begin with a single email an AI has read, a single authentication key written into a confidential document, a single file that someone in the company has uploaded.

AI returns output only; judgement and authority stay with people. Applying that obvious principle without exception to the AI that does the inspecting, and checking it on our own systems every week, is our defence.

Questions we often receive from practitioners

What kinds of security incident involve generative AI?

They fall into three routes, depending on where the AI sits. In the first, attackers use AI themselves, dividing the stages of an attack from reconnaissance to exfiltration among AI agents. In the second, attackers target the AI a company has deployed, exploiting the fact that AI cannot tell commands embedded in text from the instructions it was actually given, and steering it through an incoming email or web page. The third is shadow AI: employees' use of external AI services carries personal data and authentication keys outside the company's control.

Is it safe to let AI review code for vulnerabilities?

That depends on how the reviewing AI's permissions are designed. In one case, review agents running in a CI environment read instructions written into the title of a pull request as commands, and published the API keys and tokens in their own environments. The cause was that they had been granted program-execution rights and secrets that reviewing code never required. Nexgen Japan confines its inspector to reading and searching, gives it no production secrets and masks its work logs in multiple layers. People keep the decision to fix, and only human-approved changes reach production.

Can smaller companies afford AI-driven code inspection?

Yes. The cost depends on the volume and frequency of the code inspected, not on the size of the company; in Nexgen Japan's experience, one inspection of around 100 files costs between $0.10 and $0.20. The real barriers are not cost but the design of permissions and checkpoints, inspector included, and the discipline of ranking the holes found and fixing them continuously. That is the work of the AI Steward, who reviews AI output and decides whether to accept, correct or stop it.

Get in Touch

We design AI agent permissions and checkpoints alongside implementation

Nexgen Japan helps organisations implement and govern AI as a single effort, including establishing AI governance over generative AI application development. For advice on designing AI agent permissions and checkpoints, appointing and developing AI Stewards, or building a rolling inspection cycle for your own code, please get in touch through our contact form.

Contact us →
ChronicleKeynote at ITmedia Security Week 2026 Summer: There Is No Ctrl+Z in Reality — AI Stewards in the Age of Physical AI MediaPublished by WEF: After ChatGPT, 'Physical AI' Is Next. Can We Afford Failures in the Real World? CompassWaymo、加州の新規制で高速道路走行を停止

References and Key Sources

(16)
  1. Anthropic, "Detecting and countering misuse of AI: September 2026", September 2026
  2. Anthropic, "Investigating three real-world incidents in our cybersecurity evaluations", 30 July 2026
  3. Microsoft, "2026 Microsoft Digital Defense Report", 1 October 2026
  4. Microsoft Threat Intelligence, "Securing CI/CD in an agentic world: Claude Code Github action case", 5 June 2026
  5. Aonan Guan, "Comment and Control", published 15 April 2026; Cloud Security Alliance, "Comment and Control: When a GitHub Issue Steals CI Secrets", August 2026
  6. OWASP, "Top 10 for Agentic Applications", 9 December 2025; "Top 10 for LLM Applications 2026", 4 August 2026
  7. Chroma Research, "Context Rot: How Increasing Input Tokens Impacts LLM Performance", 2025
  8. Google Cloud / Mandiant, "AI risk and resilience in 2026: A Mandiant special report", 2026
  9. Semperis, "2026 Ransomware Risk Report", 14 October 2026
  10. National Police Agency, "Cyberspace threats in the first half of 2026", 10 September 2026 (in Japanese)
  11. Panasonic Operational Excellence Co., Ltd., "Apology and notice regarding the upload of personal data to an external generative AI service", 23 October 2026 (in Japanese)
  12. RIZAP Inc., public statement, 3 September 2026 (in Japanese)
  13. Barracuda, "Threat Spotlight: Email attacks target humans and AI in the same message", 7 October 2026
  14. Zenity Labs, "AgentCorruption", 8 October 2026
  15. Ministry of Economy, Trade and Industry, "AI Guidelines for Business"
  16. Nexgen Japan, "Job description sample: AI Steward", Nexgen Report AI Governance Series
Ariki Ono
CEO & Logistics AI Architect

Led AI, digital transformation and supply-chain reform at major IT firms and global financial institutions. Advises enterprises by combining generative-AI adoption strategy with organisational change, whilst engaging in talent training, global research and study.