A large-scale canary-token experiment shows that data submitted to AI tools does not stay where users expect — a finding with direct implications for organisations rolling out copilots, chatbots and retrieval pipelines.

A security researcher has spent recent weeks stuffing roughly two million unique, trackable secrets into prompts sent through the AI tooling ecosystem, then watching where those planted "canaries" turned up. The results, reported by Ars Technica, were described as a privacy disaster: data submitted in confidence did not remain confined to the place a user would reasonably expect it to stay.

The technique is deliberately simple. Each prompt carries a unique string — a canary token — that can be searched for later. If the string reappears anywhere downstream — in a log, a stored index, a provider's own records, or a third-party processor's system — it has been "sighted," and the route it travelled can be mapped. Rather than arguing about privacy policies, the method produces a binary answer: did this data leave, or didn't it?

Ars Technica's report frames the findings as evidence that the AI supply chain, not any single model, is the weak point. Data does not stop at the chat window: it moves through a succession of logging, retrieval and third-party processing steps, and each hand-off is a point where information can persist or travel further than intended.

The experiment's headline figure — two million planted canaries — measures the scale of the test rather than a leakage rate. The source summary states plainly that the canaries proved data does not stay private, though it does not quantify what proportion of the planted secrets surfaced, or where. Organisations reading the study should treat the methodology, not a specific percentage, as the takeaway.

For enterprises in Hong Kong and elsewhere evaluating AI adoption, that distinction matters more than any vendor's terms of service. Contract terms describe intent; a planted canary describes behaviour. Where the two diverge, the canary is the more reliable witness.

That gives security teams a practical template to copy. Before an AI tool is connected to real data, plant your own canaries — synthetic customer records, marked documents, distinctive identifiers — and trace where they surface across the stack. Audit every layer in the path, not just the visible vendor: how data is logged, what is indexed for retrieval, which plug-ins or external processors sit between the prompt and the outcome. Whatever data-protection and sector rules already apply to your organisation should be checked against what the canary test actually shows, not against what the contract promises. The same discipline applies to the deployment itself: a claim about where data goes is only as good as the evidence behind it.

Not every organisation needs to plant two million secrets. A few dozen well-chosen canaries, deployed before rollout, would tell most teams more about their real exposure than a month of reading privacy policies.


一項大規模 canary token 實驗顯示,提交給 AI 工具的資料並未留在用戶預期的位置——這項發現對部署 copilot、chatbot 及 retrieval pipeline 的機構有直接影響。

一名安全研究人員在過去數星期,向透過 AI 工具生態系統發送的提示詞(prompt)中,塞入約兩百萬個獨一無二、可追蹤的秘密資料,然後監察這些植入的「金絲雀」最終出現在哪裡。Ars Technica 報道的結果被形容為一場私隱災難:用戶滿懷信心提交的資料,並沒有如用戶合理預期般局限於原地。

這項技術刻意從簡。每條提示詞都載有一條獨特字串——即 canary token——日後可供搜尋。若該字串在下游任何位置再度出現——無論是 log、已儲存的 index、供應商自身的紀錄,或第三方處理商的系統——即代表已被「目擊」,其傳播路徑亦可被描繪出來。此方法毋須爭拗私隱政策,而是直接得出是非答案:這批資料究竟離開了,還是沒有?

Ars Technica 的報導將研究結果定位為一項證據,指出 AI 供應鏈——而非任何單一模型——才是弱點所在。資料不會止步於聊天視窗:它會經過一連串的 logging、retrieval 及第三方處理步驟,而每一個交接點,都是資料可能停留或進一步流傳至超出原定範圍之處。

實驗的焦點數字——兩百萬個植入的金絲雀——衡量的是測試規模,而非洩漏率。原始資料摘要明確指出金絲雀證明資料並非私密,但沒有量化已植入秘密中有多大比例浮現,亦沒有說明浮現於何處。閱讀該研究的機構,應把方法論而非某個特定百分比視為核心收穫。

對於香港及其他地區正評估採用 AI 的企業而言,這個分別比任何供應商的服務條款更為重要。合約條款描述的是意圖;植入的金絲雀反映的是行為。當兩者出現落差時,金絲雀才是更可靠的見證。

這為安全團隊提供了一個可套用的實用模板。在 AI 工具連接真實資料之前,先自行植入金絲雀——合成的客戶紀錄、加上標記的文件、具辨識度的識別碼——然後追蹤它們在整個技術堆疊中出沒的位置。審計路徑上的每一個層面,而不僅是表面上的供應商:資料如何被記錄、哪些內容被建立索引以供 retrieval、提示詞與最終結果之間夾雜了哪些插件或外部處理商。現行適用於貴機構的資料保護及行業規定,應對照 canary 測試的實際結果來檢視,而非對照合約上的承諾。同一原則亦適用於部署本身:關於資料去向的說法,只與背後的證據一樣可靠。

並非每個機構都需要植入兩百萬個秘密。數十個精心挑選、在推出前部署的金絲雀,已能讓大多數團隊比閱讀一個月私隱政策更清楚了解自身的實際風險暴露。

新聞來源 / Original News Source