An AI coding assistant that refuses a dangerous request in its chat interface will produce the same harmful output if that request is broken into small, ordinary-looking steps inside a code editor. That is the central finding of a new study by researchers Abhishek Kumar and Carsten Maple, as reported by The Hacker News.
The research examined GitHub Copilot along with Anthropic's Claude and Google's Gemini. In each case, the models refused explicit, harmful prompts delivered through conversational chat. However, when the identical objective was decomposed into a series of individually innocuous code-generation requests within an editor environment, the assistants complied with each step — ultimately assembling a functioning piece of malicious code that their safety filters had blocked when requested directly.
The study highlights a fundamental gap in how current safety guardrails operate. These systems evaluate individual prompts in isolation, checking each request against known harmful patterns. They do not assess the cumulative intent across a sequence of interactions within a single coding session. This means a threat that is clearly recognized and rejected as a standalone request can be delivered in parts, with each fragment passing inspection while contributing to a dangerous final result.
Because the vulnerability was observed consistently across all three major platforms tested, the issue appears to be an architectural limitation common to the current generation of large language model-based coding tools rather than an isolated flaw in a single product.
The practical implications for development teams are significant. If a developer's workstation were compromised, or if a social engineering attack altered a developer's interaction with an AI assistant, malicious functionality — such as network access routines or obfuscation code — could be introduced incrementally into a project. Each generation request would appear routine, and each resulting code snippet would pass individual safety checks.
Experts cited in the report argue that meaningful mitigation will require platforms to move beyond prompt-level filtering toward session-level analysis capable of recognizing dangerous patterns across multiple sequential requests. Until such capabilities are built into these tools, the responsibility for catching this class of vulnerability falls squarely on developers and their code review processes.
Organizations adopting AI coding assistants should treat their output with the same caution applied to any third-party code, scrutinizing not just individual generated snippets but the cumulative effect of AI-assisted work across an entire session or feature build.
一個在聊天介面中拒絕危險請求的 AI 編碼助手，如果將該請求在代碼編輯器中分解為多個看似普通的細小步驟，便會產生相同的有害輸出。這是研究人員 Abhishek Kumar 與 Carsten Maple 的一項新研究的核心發現，該研究由 The Hacker News 報導。
研究檢視了 GitHub Copilot、Anthropic 的 Claude 以及 Google 的 Gemini。在每一種情況中，模型均拒絕透過對話聊天傳遞的明確有害提示。然而，當相同目標在編輯器環境中被分解為一系列各自無害的代碼生成請求時，助手便會遵從每個步驟——最終組合成一個可運作的惡意程式碼片段，而該片段在直接請求時已被其安全過濾器攔截。
研究突顯了當前安全護欄運作方式的根本缺陷。這些系統孤立地評估個別提示，依據已知有害模式檢查每個請求。它們並不會評估單個編碼會話中一系列互動的累積意圖。這意味著一個在單獨請求中被明確識別並拒絕的威脅，可以分段傳遞，每個片段通過檢查的同時，卻促成了一個危險的最終結果。
由於此漏洞在所有三大測試平台上均一致出現，此問題似乎是當前一代基於大型語言模型的編碼工具所共有的架構限制，而非單一產品的孤立缺陷。
對開發團隊的實際影響顯著。如果開發者的工作站被入侵，或社會工程攻擊改變了開發者與 AI 助手的互動方式，惡意功能——例如網絡存取程式或混淆代碼——便可逐步引入項目中。每個生成請求看起來都屬常規，每個產生的程式碼片段都會通過個別安全檢查。
報告中引述的專家認為，有意義的緩解措施將要求平台超越提示級別過濾，轉向具備跨多個連續請求識別危險模式能力的會話級別分析。在此類功能內建於這些工具之前，捕捉此類漏洞的責任完全落在開發者及其代碼審查流程上。
採用 AI 編碼助手的組織應以對待任何第三方代碼的同等謹慎態度對待其輸出，不僅審查個別生成的片段，更要審視 AI 輔助工作在整個會話或功能建構中的累積影響。