top of page
検索

2025 AI Incident White Paper

This document is the revised edition of a white paper that structurally organised AI‑related incidents reported primarily during 2025. The earlier version lost points because the strength of evidence and the format of the evidence bundles were sometimes inconsistent. The following improvements have been applied:


* **Evidence Strength (ES) label** – Each case now carries a three‑level indicator of factual strength. **A** = recognition by the affected party or an official statement; **B** = multiple reliable media reports; **C** = a single report or a claim requiring continued verification.

* **Unified B2 (missing evidence bundle) template** – For every case we list the elements of the evidence bundle using a common template — `artifact_id / actor_id / timestamp / decision / reason_code / input_ref / output_ref / hash / signature(optional)` — and indicate which elements were missing. This makes it clear what proof was absent.

* **Case‑to‑reference mapping** – Footnotes connect each case to its sources so that readers can easily trace claims back to the underlying reporting or official statements.




----------------------------------------------------------------------

EXECUTIVE SUMMARY

----------------------------------------------------------------------

Purpose

- To structurally organise 12 AI-related incidents (primarily 2025) under a single “responsibility evaporation” template, and to state the minimum implementable requirements: a stopping boundary (fail-closed) plus a unified evidence bundle.


Intended readers

- Executives / Audit & Legal

- SRE / Engineering owners of production changes and operations

- AI providers and deployers handling high-risk decisions


How to use this white paper (fast path)

1) Pick the closest case (domain / risk / operating mode).

2) Compare your logging/controls against B2 (Missing evidence bundle) and identify gaps.

3) Implement B4 (Minimum repair). If evidence is missing, stop the action (fail-closed).


Key point (what this document does / does not claim)

- This document does NOT argue “AI is inherently dangerous.”

- It shows a repeatable structure: when the stopping boundary and evidence bundle are not pre-fixed, responsibility evaporates regardless of whether AI is involved.

- Across 12 cases, the root is not “insufficient evaluation,” but fail-open pathways created by unfixed boundaries and unfixed approval/reference subjects.


Minimum implementable requirements (3)

1) Mandate the unified evidence bundle:

artifact_id / actor_id / timestamp / decision / reason_code / input_ref / output_ref / hash / signature(optional)

2) Halt on missing or inconsistent evidence (fail-closed):

Do not execute actions (summarise / deny / publish / issue / deploy / distribute / patch, etc.) without the bundle.

3) Fix the subject of approval (diff / reference set):

Bind approvals to a diff-hash or deterministically defined reference set; unsigned subjects are non-executable.


10-minute checklist

- Is every approval bound to a concrete diff-hash (or deterministic reference set)?

- Are input_ref and output_ref linked for “reference” actions (view / summarise / generate)?

- Is there any pathway where the system proceeds when evidence is missing (fail-open)?


Expected effect

- Prevents post-incident responsibility drift by fixing actors, decisions, reasons, and evidence.

- Enables compliance/audit on the basis of verifiability (evidence), not narratives (explanations).


----------------------------------------------------------------------


The revised cases are presented below.


----------------------------------------------------------------------

CASE 1) Microsoft 365 Copilot: Confidential mail summarisation bug (DLP/label bypass)

----------------------------------------------------------------------

**ES = A**


F:

- A bug in Microsoft 365 Copilot allowed the service to summarise emails bearing confidentiality labels. The bug bypassed Data Loss Prevention (DLP) and sensitivity labels, meaning that confidential content could be read through the AI summary channel【1】.

- Microsoft acknowledged the issue in a service health notification, stating that a code problem caused the Copilot chat service to include items in the Sent Items and Drafts folders—even if they were labelled confidential—and that a fix was deployed【2】.


B0:

- Confidential information is expected to be protected not only during viewing but also when passed through AI summarisation pipelines; labels and DLP are assumed to apply everywhere.


B1:

- The AI summarisation pathway did not perform label/DLP consistency checks and therefore proceeded to access content that should have been blocked; in effect, the AI route was fail‑open.


B2 (missing evidence bundle):

- **artifact_id** (identifier for each referenced email)

- **actor_id** (identifier of the Copilot service module)

- **timestamp** (time of access)

- **decision** (whether a summary was generated)

- **reason_code** (result of label and DLP evaluation)

- **input_ref** (hash of the source message)

- **output_ref** (hash of the generated summary)

- **hash** (composite hash for tamper detection)

- **signature** (audit signature, optional)


  Because these elements were not logged together, it was difficult to determine after the fact which confidential messages were accessed by the AI.


B3:

- The boundary between the control layer (M365) and the AI summarisation layer (Copilot) was broken; administrators could not verify the assumption that labels and DLP apply across both layers. Responsibility therefore became unclear.


B4 (minimum repair):

- Require the unified evidence bundle for every AI access so that search, view and summary scopes are recorded deterministically.

- Reject any summary request if a label inconsistency is detected (fail‑closed).

- Store evidence bundles in a format that links the reference set, the policy decision and the generated output to enable auditing.


C:

- **Shows**: When control and AI reference layers are inconsistent, the responsibility for protecting secrets evaporates.

- **Does not show**: That AI systems are inherently dangerous.

- **Requires**: Evidence of the reference set and the ability to stop summarisation when labels do not match.


----------------------------------------------------------------------

CASE 2) AWS: Kiro AI‑related outage / change‑management breakdown

----------------------------------------------------------------------

**ES = B**


F:

- At the end of 2025, AWS’s cost‑management service (Cost Explorer) suffered a prolonged outage. Anonymous sources told the Financial Times that the outage lasted 13 hours because engineers allowed the “Kiro” AI coding assistant to carry out changes【3】. Amazon countered that the interruption was caused by misconfigured access controls, that AI tools were not involved and that only the cost‑explorer service in one region was affected【4】【5】.


B0:

- Changes to production systems, whether made by humans or AI, are supposed to follow change‑management procedures that include review, approval and a roll‑back plan.


B1:

- The unit of approval drifted toward “who said OK” rather than “what diff was approved.” When the subject of approval is not tied to a specific code or configuration hash, execution can proceed in a fail‑open manner.


B2 (missing evidence bundle):

- **artifact_id** (hash of the code or configuration diff being deployed)

- **actor_id** (identifier of the Kiro tool or engineer)

- **timestamp** (time of approval and execution)

- **decision** (approve/reject/roll back)

- **reason_code** (basis for the approval, e.g., risk assessment)

- **input_ref** (state before the change)

- **output_ref** (state after the change)

- **hash** (hash linking approval and execution logs)

- **signature** (reviewer’s signature)


  Without linking the diff hash to the approval signature, it is difficult to verify after the fact what was approved and by whom.


B3:

- When SREs, developers and AI tool providers do not agree on what exactly is being approved, it is impossible to assign responsibility if something goes wrong.


B4 (minimum repair):

- Bind approvals to a diff hash and require a signature; unsigned diffs cannot be executed (fail‑closed).

- Automatically stop and roll back if the hash in the execution log does not match the approved hash.


C:

- **Shows**: Change management that does not fix the subject of approval allows responsibility to evaporate, regardless of whether AI is involved.

- **Does not show**: That AI was the root cause of the outage.

- **Requires**: Hash‑based diff binding and halting execution when it lacks a signature.


----------------------------------------------------------------------

CASE 3) Waymo: Service disruption and congestion during San Francisco power outage

----------------------------------------------------------------------

**ES = B**


F:

- During a major power outage in San Francisco in December 2025, traffic lights failed and Waymo’s autonomous taxis stopped at intersections, causing congestion. Several vehicles sat at junctions with hazard lights flashing and traffic built up behind them【6】.

- Waymo suspended service during the blackout for safety. A company spokesperson said remote assistance was delayed due to the scale of the outage, so vehicles remained stationary longer than usual【7】.


B0:

- Operational procedures assume that when external infrastructure such as traffic lights and communications fails, vehicles will stop safely and remote support will be provided.


B1:

- While choosing to stop was the safe decision, there was no predefined procedure for where to pull over or how to mitigate traffic impact. The stop boundary was therefore undefined and the system behaved in a fail‑open manner.


B2 (missing evidence bundle):

- **artifact_id** (vehicle ID and location log)

- **actor_id** (autonomy software module or remote operator)

- **timestamp** (times of outage detection, stopping and resuming)

- **decision** (stop, pull over, resume)

- **reason_code** (external conditions such as power outage)

- **input_ref** (external infrastructure status)

- **output_ref** (record of stop location and vehicle behaviour)

- **hash** (hash linking the above logs)

- **signature** (operations manager’s signature)


  Without an integrated log of power status, vehicle decisions and remote intervention, it is hard to reconstruct who did what and when.


B3:

- Municipalities, infrastructure operators and the fleet operator all lacked a fixed responsibility for clearing traffic. Responsibility evaporated among them.


B4 (minimum repair):

- Predefine safe pull‑over locations and procedures for external failures and implement a fail‑closed retreat when remote assistance is delayed.

- Share evidence of external conditions and record the sequence of outage detection and retreat decisions according to the unified template.


C:

- **Shows**: When external assumptions collapse and retreat procedures are undefined, responsibility for congestion evaporates.

- **Does not show**: A rejection of autonomous driving in general.

- **Requires**: Evidence of external conditions and a precommitted retreat procedure.


----------------------------------------------------------------------

CASE 4) UnitedHealth Group: Lawsuit over nH Predict and care denial

----------------------------------------------------------------------

**ES = B**


F:

- In the United States, a class action alleges that UnitedHealth Group’s AI prediction tool “nH Predict” was used to deny post‑acute care payments to elderly patients. Plaintiffs claim that the AI has a high error rate and supplanted physicians’ determinations【8】【9】.

- In February 2025, a federal court declined to dismiss certain breach‑of‑contract claims and allowed the case to proceed, finding that Medicare rules did not pre‑empt the plaintiffs’ claims【8】.


B0:

- Insurers present their processes as involving human physicians making the final decision, with AI serving only as a guide.


B1:

- In practice, the AI’s output carried binding weight and denials were often overturned only on appeal. Denial became fail‑open: even weak evidence triggered denial.


B2 (missing evidence bundle):

- **artifact_id** (identifier for each claim)

- **actor_id** (nH Predict model version and reviewer)

- **timestamp** (AI recommendation, human override and notification times)

- **decision** (approval, denial, overturn)

- **reason_code** (AI score, thresholds, policy provisions)

- **input_ref** (patient medical data referenced)

- **output_ref** (AI recommendation and final human decision)

- **hash** (hash of the decision path)

- **signature** (physician’s signature)


  Without linking denial grounds and human overrides in a single evidence bundle, it is impossible to determine responsibility.


B3:

- Responsibility evaporates among insurers (decision makers), model providers and healthcare providers when AI outputs dominate but are not fully documented.


B4 (minimum repair):

- Tie every denial to the unified evidence bundle; denials lacking adequate evidence should not be executed.

- If evidence is insufficient, halt AI‑driven denial (fail‑closed) and require human review.

- Whenever a denial is overturned, record the grounds and reasoning within the same evidence bundle.


C:

- **Shows**: High‑risk medical decisions without strong evidence bundles lead to structural responsibility evaporation.

- **Does not show**: A blanket rejection of AI in healthcare.

- **Requires**: Denial decisions anchored in evidence bundles and halted when evidence is insufficient.


----------------------------------------------------------------------

CASE 5) AI hiring screening: discrimination lawsuits (e.g., Sirius XM)

----------------------------------------------------------------------

**ES = B**


F:

- Several lawsuits challenge AI‑based hiring tools for discriminatory impact. In **Harper v. Sirius XM**, the plaintiff contends that an AI screening tool trained on past hiring data downgraded Black applicants by using proxies such as education, zip code and employment history【10】. In **Mobley v. Workday**, older applicants allege age discrimination; the court conditionally certified an Age Discrimination in Employment Act collective【11】. U.S. states are introducing regulations requiring proactive bias testing, record‑keeping and even third‑party liability for AI vendors【12】.


B0:

- Employers have a duty of fairness in hiring and must provide explanation and auditability; AI systems must meet the same obligations.


B1:

- Many companies operate AI tools without meaningful audits, relying on vendor indemnities. With no fixed point of responsibility, operation is fail‑open.


B2 (missing evidence bundle):

- **artifact_id** (record ID for each candidate evaluation)

- **actor_id** (AI model version and recruiter)

- **timestamp** (times of screening, model updates and notifications)

- **decision** (pass/fail)

- **reason_code** (scores or feature contributions)

- **input_ref** (résumés and applicant data)

- **output_ref** (AI score and final decision)

- **hash** (hash of the logic and threshold update history)

- **signature** (auditor or HR officer’s signature)


  Without such logs it is impossible even to determine whether discrimination occurred, much less assign responsibility.


B3:

- Responsibility is dispersed among the employer, the model vendor and auditors. Contracts often blur the lines, causing responsibility to evaporate.


B4 (minimum repair):

- Prioritise verifiability over full explainability; store evaluation logs, thresholds and model update histories as evidence bundles.

- If auditing cannot be performed, suspend AI operation (fail‑closed) and revert to human processes.


C:

- **Shows**: Hiring AI that cannot be audited allows responsibility to evaporate.

- **Does not show**: That all AI‑enabled hiring should be banned.

- **Requires**: Verifiable evidence bundles and suspension of systems that cannot be audited.


----------------------------------------------------------------------

CASE 6) Japan: Kaikatsu CLUB cyberattack assisted by generative AI

----------------------------------------------------------------------

**ES = C**


F:

- In a cyberattack on the Kaikatsu CLUB net‑café chain, Tokyo police arrested a high‑school student who had allegedly obtained about 7.25 million membership records. Investigators said he used the ChatGPT generative AI to help create the attack code and send fraudulent commands to the server【13】.


B0:

- Service operators must deploy authentication, rate limiting, anomaly detection and log preservation. When an attack occurs, evidence must be retained to support investigation and remediation.


B1:

- The attacker automated the intrusion and evaded detection, allowing a large data theft. Defences remained fail‑open.


B2 (missing evidence bundle):

- **artifact_id** (log IDs for login attempts and commands)

- **actor_id** (source IP or device fingerprint)

- **timestamp** (times of each unauthorised access attempt)

- **decision** (permit/deny/alert)

- **reason_code** (anomaly detection rule and threshold)

- **input_ref** (request payload)

- **output_ref** (server response)

- **hash** (hash to prevent log tampering)

- **signature** (seal by administrators)


  Without consistent preservation of login and anomaly logs, it is impossible to reconstruct the intrusion path and defensive failures.


B3:

- The boundary between the operator (defence), the identity verification contractor and the attacker was blurred. Insufficient evidence on the operator side became the point where responsibility evaporated.


B4 (minimum repair):

- Implement automatic halting or rate limiting when thresholds are exceeded, blocking suspicious access (fail‑closed).

- Standardise log formats and preserve evidence bundles according to the unified template.

- Strengthen authentication with multi‑factor authentication to counter automated attacks.


C:

- **Shows**: Capability amplification without upgraded evidentiary requirements leads to evaporating responsibility.

- **Does not show**: That AI is the cause of crime per se.

- **Requires**: Additional evidence bundles and halting when detection fails.


----------------------------------------------------------------------

CASE 7) Amazon Q Developer / VS Code extension: supply‑chain tampering allegations

----------------------------------------------------------------------

**ES = C**


F:

- Reports emerged that Amazon’s AI coding assistant “Q Developer” extension for Visual Studio Code had been tampered with and contained destructive commands. A hacker using an unverified GitHub account submitted a pull request, and version 1.84.0 shipped with code that attempted to delete user data and cloud resources. Amazon quickly mitigated the issue, released version 1.85 and said no customer resources were impacted【14】. Experts highlighted supply‑chain risks and the need for immutable pipelines, code signing and anomaly detection【15】.


B0:

- Distribution packages are expected to be signed and verified such that tampering prevents deployment; provenance must be assured before production use.


B1:

- In this case, authenticity checks in the update and distribution channels were lax, and a tampered version was temporarily distributed. The installation pipeline was fail‑open.


B2 (missing evidence bundle):

- **artifact_id** (hash or version ID of each package)

- **actor_id** (committer identity and build pipeline)

- **timestamp** (times of commit, build and distribution)

- **decision** (verification success/failure)

- **reason_code** (verification result or warnings)

- **input_ref** (source repository)

- **output_ref** (distributed package)

- **hash** (hash of SBOM or signature)

- **signature** (developer and build server signatures)


  Without linking distribution hashes, signature verification logs and update paths, it is impossible to determine at which stage tampering occurred.


B3:

- Boundaries blurred among signers (code providers), marketplace operators (distributors) and user organisations (deployment). It became unclear who should prove what, leading to evaporated responsibility.


B4 (minimum repair):

- Mandate signature and provenance verification; mismatches must trigger immediate halting (fail‑closed).

- Preserve software bills of materials (SBOM) and attestation as evidence bundles, and require deployment verification logs before production use.


C:

- **Shows**: Regardless of AI, if supply‑chain authenticity is not fixed, responsibility evaporates.

- **Does not show**: That AI development tools are inherently dangerous.

- **Requires**: Provenance evidence bundles and halting on unverified components.


----------------------------------------------------------------------

CASE 8) FortiGate: Large‑scale intrusions and AI‑augmented attacker narrative

----------------------------------------------------------------------

**ES = B**


F:

- In early 2026 Amazon Threat Intelligence observed a Russian‑speaking threat actor who used generative AI services to compromise over 600 FortiGate devices across 55 countries【16】【17】. The campaign did not exploit a FortiGate vulnerability; instead it targeted exposed management ports and weak single‑factor credentials. AI tools automated scanning, intrusion and the creation of target lists, enabling attackers with limited skills to operate at scale【16】.


B0:

- Vendors are expected to provide patches after disclosing vulnerabilities, and operators must apply those patches and manage exposure. Proof of patch application is part of the pre‑commitment.


B1:

- Patches were published but there was no way to prove whether they had been applied. Operations continued without evidence of patching, leaving devices exposed. The process was fail‑open.


B2 (missing evidence bundle):

- **artifact_id** (log ID for each device’s patch application)

- **actor_id** (vendor, operator or managed service provider)

- **timestamp** (times of vulnerability disclosure, patch release and application)

- **decision** (applied/not applied/deferred)

- **reason_code** (reasons for not applying or delaying a patch)

- **input_ref** (asset inventory)

- **output_ref** (post‑patch version information)

- **hash** (hash of the application evidence)

- **signature** (signature of the operations responsible person)


  Without an evidence bundle linking vulnerability disclosure, patch release, patch application and the timeline of the intrusion, it is hard to assign responsibility.


B3:

- Between the vendor (providing fixes), the operator (applying patches) and managed service providers (acting on behalf of operators), responsibility for application could not be determined without proof.


B4 (minimum repair):

- Require proof‑of‑application logs and halt exposure when patch status is unknown (fail‑closed).

- Link vulnerability disclosure, patch release, patch application and verification events into a single evidence bundle and set out preservation procedures for incident response.


C:

- **Shows**: Without proof of patch application, responsibilities after a compromise cannot be determined and therefore evaporate.

- **Does not show**: That AI‑enabled attacks were the sole cause.

- **Requires**: Patch application evidence and halting when status is unknown.


----------------------------------------------------------------------

CASE 9) Hungary: AI‑generated election video (fictitious execution scene)

----------------------------------------------------------------------

**ES = B**


F:

- In February 2026 the ruling Fidesz party’s Budapest branch posted a 33‑second election video on Facebook that cut together a blindfolded Hungarian soldier being executed on a battlefield and a little girl crying at a window, urging voters to choose “war or peace”【18】. Opposition leader Péter Magyar condemned the clip as manipulation.

- The video was reported to have been made using artificial intelligence; a senior government official did not deny this, and Reuters confirmed that Google’s AI models had helped to produce the wartime footage【18】. The EU’s proposed AI Act would require disclosure of AI‑generated political advertisements.


B0:

- Election information is subject to rules requiring disclosure of the origin, and mechanisms for correction and takedown should function during campaigns.


B1:

- The origin of the video was unclear, yet it spread widely. Because the entity responsible for correction was ambiguous, the platform operated in a fail‑open manner.


B2 (missing evidence bundle):

- **artifact_id** (video ID or URL)

- **actor_id** (producer and distributing account)

- **timestamp** (time of creation, posting and removal)

- **decision** (post allowed, deleted or corrected)

- **reason_code** (deepfake detection results or complaint rationale)

- **input_ref** (source material and models used)

- **output_ref** (hash of the video)

- **hash** (combined hash of metadata and logs)

- **signature** (platform or regulator signature)


  Missing provenance flags, AI‑generation indicators, distribution logs and timelines of correction orders meant that responsibility for removal evaporated.


B3:

- Without clear boundaries between producers, platforms and regulators, and with no procedural logs of stopping or correcting content, responsibility remained unclear.


B4 (minimum repair):

- Halt distribution of videos with unknown origin or without generation flags (fail‑closed), and allow re‑posting only after an appeal.

- Standardise the evidence bundle including origin, generation flags, distribution paths and removal/reinstatement timelines, and store it in a verifiable format.


C:

- **Shows**: In political contexts the key is not “genuineness” but the procedures for stopping and correcting; when logs are weak, responsibility evaporates.

- **Does not show**: A judgement about the rights or wrongs of any party.

- **Requires**: Provenance proof and distribution controls; content should be restricted if the evidence bundle is incomplete.


----------------------------------------------------------------------

CASE 10) Romania: Deepfake investment scam using presidential candidates (Neptun Deep)

----------------------------------------------------------------------

**ES = C**


F:

- In May 2025, ahead of Romania’s presidential election, AI‑generated videos were posted on Facebook depicting candidates George Simion and Nicușor Dan promoting a fictitious government investment called “Neptun Deep.” The videos claimed that paying 1,400 lei would yield 9,000 lei in monthly passive income【19】.

- According to Bitdefender, the scam used deepfake technology to clone the candidates’ faces and voices and redirect viewers to fake news articles and websites where they were asked to transfer money. The George Simion video was removed, but the Nicușor Dan version remained active and rapidly gained reactions【19】. Similar scams using celebrity deepfakes and cloned media brands have circulated before【20】.


B0:

- Advertising and investment solicitations are supposed to include know‑your‑customer (KYC) checks and mechanisms for reporting and correction.


B1:

- The ads continued even though the identity of the advertiser could not be confirmed. The ad distribution pipeline was fail‑open.


B2 (missing evidence bundle):

- **artifact_id** (ad ID or video URL)

- **actor_id** (advertiser or fraud group)

- **timestamp** (times of posting, reporting and removal)

- **decision** (allowed/deleted/frozen)

- **reason_code** (identity verification results or complaint details)

- **input_ref** (video training data of the real person)

- **output_ref** (hash of the deepfake video)

- **hash** (hash of KYC documents and logs)

- **signature** (signatures of the platform and advertiser)


  Without KYC information, identity verification evidence, timelines for removal requests and records of remediation, both redress and attribution evaporate.


B3:

- The weaker the identity evidence, the more responsibility drifts among fraudsters, advertising platforms and financial institutions providing redress.


B4 (minimum repair):

- Do not publish ads whose originators cannot be firmly identified (fail‑closed); commit to SLAs for identity verification and removal requests.

- Standardise evidence bundles for reports, takedowns and restitution so that progress and decisions are recorded.


C:

- **Shows**: When evidence of identity is weak, both responsibility and redress evaporate.

- **Does not show**: A general claim that no video can be trusted.

- **Requires**: Identity evidence bundles and refusal to publish when they are lacking.


----------------------------------------------------------------------

CASE 11) Singapore: GE2025 and rising AI‑generated content (regulatory operations)

----------------------------------------------------------------------

**ES = B**


F:

- After the writ of election for Singapore’s 2025 general election, the number of AI‑generated political videos on platforms such as TikTok increased sharply. During just five days, from 15–19 April 2025, 73 election‑related AI videos were detected; 11 of them contained manipulated visuals of prospective candidates【22】.

- In October 2024 Parliament amended the Elections (Integrity of Online Advertising) Bill to prohibit publishing, boosting, sharing or reposting deepfake content that misrepresents candidates. The ban applies from the issuance of the writ until the close of polling and covers AI‑generated and other digitally manipulated content. Individuals who violate the law may face fines and up to 12 months’ imprisonment; social media platforms that do not comply may be fined up to SGD 1 million【23】. Benign modifications such as beauty filters and non‑realistic content are excluded【23】.

- The Ministry of Digital Development and Information (MDDI) monitors platforms in cooperation with social‑media companies. Some videos were clearly labelled as AI‑generated while others were satire or misleading portrayals. Experts warn that deepfakes threaten public trust and electoral integrity【22】.


B0:

- Rules already exist for labelling, correction orders and immediate takedown; operations are expected to follow these procedures.


B1:

- The key issue is not whether content is AI‑generated but whether the procedures for stopping and keeping evidence are fast and transparent. Without robust logs, regulatory actions can appear arbitrary and responsibility evaporates.


B2 (missing evidence bundle):

- **artifact_id** (ID of the content in question)

- **actor_id** (poster, platform and regulator)

- **timestamp** (times of order issuance, takedown and reinstatement)

- **decision** (halt/reinstate/maintain)

- **reason_code** (basis for the order or review)

- **input_ref** (content of complaint or detection algorithm output)

- **output_ref** (execution log or notification content)

- **hash** (hash to prevent log tampering)

- **signature** (signature of the authority issuing the order)


  If there is no record of the grounds for an order, the content ID, the execution log or staged enforcement measures, the legitimacy of the operation cannot be established.


B3:

- Boundaries among candidates, platforms and authorities blur when there is no clarity on who must prove what. Responsibility evaporates.


B4 (minimum repair):

- Introduce a two‑stage process in which suspicious content is temporarily halted (fail‑closed) and reinstated only after evidence review.

- Bundle all orders and executions according to the unified template and develop a mechanism to publish logs where appropriate.


C:

- **Shows**: The substance of the system is the procedure for halting content; when logs are weak, responsibility evaporates.

- **Does not show**: A value judgement about the regulation itself.

- **Requires**: A two‑stage process of temporary halting and evidence review.


----------------------------------------------------------------------

CASE 12) Japan: AI‑enabled illegal phone contracts / unauthorised access (teen arrests)

----------------------------------------------------------------------

**ES = C**


F:

- In February 2025 the Tokyo Metropolitan Police arrested three teenage boys aged 14–16. They allegedly used the ChatGPT AI chatbot to develop a program that automatically logged into Rakuten Mobile’s system using stolen IDs and passwords, obtained about 2,500 mobile phone subscriptions over six months and sold them for approximately ¥7.5 million in cryptocurrency【24】. The program automated procedures such as entering the stolen credentials【24】.


B0:

- Telecommunications contracts require identity verification (KYC), fraud detection and controls on issuance volume. Multiple applications should trigger detection and suspension.


B1:

- The teenagers automated and multiplied subscription requests, yet issuance did not stop. The issuance process was fail‑open and lacked fraud detection.


B2 (missing evidence bundle):

- **artifact_id** (ID of each application and contract)

- **actor_id** (applicant, reviewer and system module)

- **timestamp** (times of application, review, issuance and suspension)

- **decision** (approve/refuse/pending)

- **reason_code** (identity verification result or fraud detection reason)

- **input_ref** (submitted KYC documents and identifiers)

- **output_ref** (contract or subscription information issued)

- **hash** (hash of each process)

- **signature** (signature of the person responsible for review)


  Without KYC evidence, logs of multiple applications, logs of suspension decisions or post‑issuance remediation logs, the issuer’s responsibility evaporates.


B3:

- Boundaries between the issuer (responsible for issuing contracts), the KYC contractor and the fraudsters were not fixed; without a defined stopping boundary, responsibility drifted.


B4 (minimum repair):

- Hold issuance until doubt is resolved (fail‑closed) and standardise evidence bundles for each stage of application, review and issuance.

- When thresholds for multiple applications or suspicious correlations are exceeded, automatically place the application on hold for human confirmation.


C:

- **Shows**: Without a stopping boundary in the issuance process, fraud can become industrialised and responsibility evaporates.

- **Does not show**: A general claim that AI increases crime.

- **Requires**: Evidence bundles for issuance and the ability to halt when doubts arise.


----------------------------------------------------------------------

CROSS‑CASE SYNTHESIS: Responsibility evaporation patterns (12 cases)

----------------------------------------------------------------------

Across the 12 cases, responsibility evaporates in the following structural patterns, regardless of whether AI is involved:


P1 Control is bypassed only on the AI pathway (Case 1)

P2 The subject of approval is not fixed (Case 2)

P3 Responsibility for operations collapses when external assumptions fail (Case 3)

P4 High‑risk decisions lack strong evidence bundles (Case 4)

P5 Contracts and indemnities render systems unauditable (Case 5)

P6 Legacy designs cannot cope with capability amplification (Cases 6 and 12)

P7 Supply‑chain authenticity is not fixed (Case 7)

P8 There is no proof of patch application (Case 8)

P9 Provenance flags and corrective procedure logs are weak (Cases 9 and 11)

P10 Identity proof is weak (Case 10)


----------------------------------------------------------------------

FINAL CONCLUSION (one sentence)

----------------------------------------------------------------------

Unless we fix the stopping boundary (fail‑closed) and pre‑define the evidence bundle (`artifact_id / actor_id / timestamp / decision / reason_code / input_ref / output_ref / hash / signature`), responsibility will evaporate structurally, regardless of the presence of AI.


----------------------------------------------------------------------

REFERENCES

----------------------------------------------------------------------

[2] Office 365 IT Pros blog: Copilot summarises emails with Confidential labels due to code errorhttps://office365itpros.com/2026/02/13/dlp-policy-for-copilot-bug/#:~:text=An%20embarrassing%20security%20glitch%20appeared,don%E2%80%99t%20appear%20to%20be%20affected

[4] Amazon security blog: outage caused by misconfigured access controls, not AIhttps://www.aboutamazon.com/news/aws/aws-service-outage-ai-bot-kiro#:~:text=We%20want%20to%20address%20the,AI%20as%20the%20story%20claims

[7] Mission Local: Waymo suspended service during city‑wide blackout for safetyhttps://missionlocal.org/2025/12/sf-waymo-halts-service-blackout/#:~:text=%E2%80%9CWe%20have%20temporarily%20suspended%20our,%E2%80%9D

[8] DLA Piper AI Outlook: lawsuit alleges nH Predict used to deny post‑acute care; court allows breach of contract claimshttps://www.dlapiper.com/en/insights/publications/ai-outlook/2025/lawsuit-over-ai-usage-by-medicare-advantage-plans-allowed-to-proceed#:~:text=The%20US%20District%20Court%20for,on%20the%20facts%20at%20issue

[11] Lathrop GPM: Mobley v. Workday receives class certification; Harper case alleges automatic rejection of Black applicantshttps://www.jdsupra.com/legalnews/lawsuits-alleging-systemic-bias-in-ai-5418886/#:~:text=In%C2%A0Mobley%20v,reliance%20on%20biased%20training%20data

[13] The Japan Times: High‑school student arrested for Kaikatsu Club cyberattack assisted by ChatGPThttps://www.japantimes.co.jp/news/2025/12/04/japan/crime-legal/police-arrest-cyberattack-net-cafe/#:~:text=Dec%204%2C%202025

[15] CSO Online: experts highlight supply‑chain risks and need for code‑signing and immutable pipelineshttps://www.csoonline.com/article/4027963/hacker-inserts-destructive-code-in-amazon-q-as-update-goes-live.html#:~:text=Exploiting%20AI%20coding%20tools

[16] The Hacker News: AI‑augmented threat actor compromised over 600 FortiGate devices by exploiting exposed ports and weak credentialshttps://thehackernews.com/2026/02/ai-assisted-threat-actor-compromises.html#:~:text=A%20Russian,devices%20located%20in%2055%20countries

[17] Cybersecurity Dive: generative AI used to plan intrusions and parse stolen configurations, enabling large‑scale attackshttps://www.cybersecuritydive.com/news/ai-cyberattacks-fortigate-amazon/812830/#:~:text=A%20Russian,February%2C%20according%20to%20Amazon%20researchers

[18] Reuters: Hungarian opposition condemns Fidesz election video with fictitious execution scene; video made using Google’s AI modelshttps://www.reuters.com/world/hungary-opposition-condemns-fidesz-election-video-with-fictitious-execution-2026-02-19/#:~:text=,Gergely%20Gulyas%20said

[19] Bitdefender: Deepfake scam exploits Romanian presidential candidates to lure victims into fake “Neptun Deep” investmenthttps://www.bitdefender.com/en-us/blog/hotforsecurity/deepfake-scam-exploits-romanian-presidential-candidates-to-lure-victims-into-fake-neptun-deep-investment#:~:text=As%20Romania%20prepares%20for%20the,%E2%80%9D

[21] Balkan Insight: Romanian energy minister files criminal complaint over deepfake video promoting nonexistent investment platformhttps://balkaninsight.com/2024/04/12/romanian-minister-files-crime-complaint-over-deepfake-video/#:~:text=Romania%E2%80%99s%20Energy%20Minister%2C%20Sebastian%20Burduja%2C,showing%20his%20image%20and%20voice

[22] AI Law – International Review of Artificial Intelligence Law: GE2025 surge in AI‑generated election videos; 73 videos detected, 11 manipulated; MDDI monitors contenthttps://www.reviewofailaw.com/Tool/Evidenza/Single/view_html#:~:text=Following%20the%20issuance%20of%20the,manipulated%20visuals%20of%20prospective%20candidates

[23] Baker McKenzie InsightPlus: Singapore’s amended Elections (Integrity of Online Advertising) Bill prohibits publishing, sharing or boosting deepfake content about candidates during election periods; law covers AI‑generated and other manipulated content and imposes fines for individuals and platformshttps://insightplus.bakermckenzie.com/bm/technology-media-telecommunications_1/singapore-law-passed-to-ban-deepfakes-during-general-elections#:~:text=On%2015%20October%202024%2C%20Singapore%27s,deepfake%20content%20depicting%20election%20candidates

[24] The Japan Times: Three teenagers arrested for using ChatGPT to automatically log into Rakuten Mobile and obtain thousands of fraudulent phone contractshttps://www.japantimes.co.jp/news/2025/02/28/japan/crime-legal/boys-arrest-ai-subscriptions/#:~:text=Tokyo%20police%20have%20arrested%20three,made%20program%20using%20artificial%20intelligence


 
 
 

コメント

bottom of page