Key points
- Snyk’s Evo Remediation Agent passed 90.2% of upgrade tasks in the VulnBump benchmark.
- Unaided Claude Code passed only 59.6%, while adding Snyk data raised its rate to 77.5%.
- The study tested 307 real upgrades across six major package managers using production-like data.
Snyk research demonstrates that artificial intelligence agents can successfully upgrade vulnerable software dependencies in most cases without breaking the build process, though success depends heavily on the specific tools and data provided to the agents.
How it unfolded
- Snyk researchers published the VulnBump benchmark in 2026, testing AI agents on 307 real-world dependency upgrades.
- The study found that Snyk’s Evo Remediation Agent passed 90.2% of tasks, significantly outperforming unaided Claude Code.
- Analysis revealed that while keeping builds functional was common, correctly removing vulnerabilities remained the primary challenge for general-purpose agents.
Who is affected
The findings impact organizations managing open-source software supply chains, where 94% of vulnerabilities have known fixes but only 10% are resolved. System administrators and development teams using package managers like npm, pip, Maven, Gradle, Yarn, and Poetry face the ongoing difficulty of upgrading packages safely. The study highlights that even minor version updates often contain breaking changes, complicating automated remediation efforts across the 300,000 organizations monitored by Snyk Open Source.
What happens next
Teams evaluating AI for security remediation should consider that general coding assistants may not reliably remove vulnerabilities without specialized security data. The benchmark indicates that combining AI agents with precise vulnerability intelligence, such as Snyk’s recommended versions and scanning, significantly improves success rates. Purpose-built remediation agents showed the highest accuracy, suggesting that specialized tools may be more effective than general-purpose models for maintaining secure and stable software builds.
The background in plain language
Software projects rely on external libraries called dependencies. When a security flaw is found in a library, teams must upgrade to a fixed version. This process is risky because changing a library can break the application’s code. A "build break" occurs when the software fails to compile or run tests after an update. Teams usually test upgrades in isolated environments first. They check whether the new version resolves the security issue without disrupting existing functionality. Automated tools can attempt these upgrades, but they must verify both security clearance and system stability.
What to do and how to stay safe: Snyk
- Verify that any automated upgrade tool confirms the vulnerability is removed via a fresh security scan, not just by claiming success.
- Test dependency upgrades in isolated environments that mirror production to ensure builds pass and tests succeed before merging changes.
- Monitor for false negatives where agents report fixes that scans later reveal are still present, as this was a common failure mode.
- Review whether your current AI tools have access to specific vulnerability intelligence, as general models performed worse without targeted data.
General security guidance from the Circuit Observer newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
How many upgrades did the VulnBump benchmark test?
The benchmark tested 307 real upgrades across 43 repositories, covering 2,066 fixable vulnerabilities in six package managers.
Did the AI agents break the builds often?
No, all three agents kept builds functional about 93% of the time; the main difficulty was ensuring vulnerabilities were actually removed.
Which package managers were included in the study?
The study included Maven, Gradle, npm, Yarn, pip, and Poetry, which account for about 92% of production findings.
