AI Coding Tools Are Outrunning the Evidence for Their Own ROI, Says James Shore
Most developers believe AI coding assistants make them faster. James Shore wants to know by how much, what it costs, and whether the tool is being used where it actually pays off.
Writing on his Successful Software blog and republished by O'Reilly Radar with his permission, Shore frames the current enthusiasm around AI-assisted development around three questions: How much faster are you really going? What value are you getting for your money? And are there other, better ways to use AI in delivery? The piece is less a verdict on AI tools than a methodology for producing one — an argument that teams adopting copilots and code generators are scaling spending faster than they are scaling evidence.
A measurement framework, not a hot take
What distinguishes Shore's approach from the stream of opinion pieces on AI productivity is its structure. Rather than arguing that AI helps or harms, he lays out a sequence of questions a team can actually answer with data. That makes the piece unusually useful for organisations currently benchmarking rollout decisions: it converts a diffuse debate into an audit trail.
The first, and most commonly skipped, step is establishing a baseline before deployment. Without pre-adoption measurements of cycle time, throughput, and defect rates, any post-rollout figure is unfalsifiable. The second is accounting for total cost, not just per-seat licence fees. Integration effort, review overhead, training, and the reconciliation of conventions between AI-generated and hand-written code rarely appear in vendor ROI slides, yet they determine whether the net gain is positive.
A third caution concerns scope: measurement should be end-to-end, not confined to keystrokes. Time saved at the keyboard can be quietly consumed downstream — in code review queues, in rework when generated code doesn't match team conventions, in debugging behaviour the original author didn't write. Any metric that stops at the editor overstates the gain.
Shore's final question — where else can AI be deployed? — pushes teams to look beyond code generation itself, toward requirements work, testing, documentation, and maintenance, where the cost-benefit profile may differ substantially from the headline use case.
Pilot, don't roll out
For teams weighing their next procurement decision, the practical takeaways are fourfold: establish a baseline before you buy; count the total cost of ownership, not just licences; run controlled pilots rather than blanket rollouts; and evaluate AI's value across the whole delivery lifecycle rather than at the code-generation step alone.
These are modest claims, but that is their strength. In a market where vendor ROI narratives are travelling faster than the procurement due diligence meant to check them, Shore's framework offers a defensible way to structure the question. The questions are easy to ask. The discipline to answer them honestly is the harder part — and Shore's central point is that AI tooling should face the same evidentiary standard as any other infrastructure spend.
Shore's full analysis is available via O'Reilly Radar.
James Shore：AI 編碼工具的推廣速度，遠超其自身 ROI 的證據支持
大多數開發者相信 AI 編碼助手能提升效率。James Shore 想知道的是：到底快了多少、代價為何，以及這些工具是否真的被用在能產生回報的地方。
Shore 在其 Successful Software 網誌撰文，經 O'Reilly Radar 取得其授權轉載。文章圍繞三個問題剖析當前 AI 輔助開發的熱潮：你實際上究竟快了多少？花費換來了什麼價值？以及在軟件交付中，是否有其他更佳的 AI 應用方式？這篇文章與其說是對 AI 工具的判詞，不如說是一套得出判詞的方法論——其核心論點是：採用 copilot 和 code generator 的團隊，其開支擴張的速度快於證據累積的速度。
是度量框架，而非一家之言
Shore 的做法有別於連篇累牘的 AI 生產力評論文章之處，在於其結構性。他並未爭論 AI 是有益還是有害，而是列出一系列團隊真正可用數據回答的問題。這使文章對正在評估部署決策的機構格外有用：它將一場流於泛泛的辯論轉化為可追溯的審計紀錄。
第一步，也是最常被略過的一步，是在部署前建立基準線（baseline）。若沒有在採用前量度 cycle time、throughput 及缺陷率（defect rate），部署後的任何數據都無法被驗證真偽。第二步是計算總成本，而非僅計 per-seat 授權費用。整合工作量、code review 開銷、培訓，以及 AI 生成程式碼與手寫程式碼之間的規約協調，很少會出現在供應商的 ROI 簡報中，但這些因素決定了淨收益究竟是正還是負。
第三點注意事項涉及度量範圍：度量應是端到端（end-to-end），而不應局限於鍵盤操作。在鍵盤上省下的時間，可能在下游悄然被消耗——例如 code review 排隊、生成程式碼不符團隊規約時的返工、以及除錯那些原作者並未編寫的行為。任何止步於編輯器的指標，都會高估實際收益。
Shore 的最後一個問題——AI 還可以部署在哪裡？——促使團隊把目光投向程式碼生成本身之外，包括需求分析、測試、文件編寫及維護工作，這些領域的成本效益結構可能與主要應用場景大不相同。
先做 pilot，勿急於全面 rollout
對於正在權衡下一輪採購決策的團隊，實質建議有四點：購買前先建立基準線；計算 total cost of ownership，而非僅看授權費；進行受控 pilot 而非全面 rollout；以及在整個交付生命週期的層面評估 AI 的價值，而非只看程式碼生成這一環節。
這些說法看似平淡，但正因平淡而有力。在一個供應商的 ROI 敘事傳播速度，快於本應用來核查它們的採購盡職調查（due diligence）的市場中，Shore 的框架提供了一種站得住腳的方式來結構化這個問題。問題本身不難提出；難的是有紀律地誠實回答——而 Shore 的核心觀點是：AI 工具理應面對與任何其他基建開支相同的證據標準。
Shore 的完整分析可透過 O'Reilly Radar 閱讀。