DeepBinary
Can AI agents uncover deep vulnerabilities in production binaries?
DeepBinary benchmarks AI agents on real Windows system binaries.
Findings are evaluated using executable evidence, security impact, and estimated bounty.
Discovery performance
Loading discovery curves…
Leaderboard
v1Vulnerability discovery 6 configurations
Ranked by unique bug count. Higher is better.
Loading results…
How is bounty calculated?
Unique bugs are findings accepted by the benchmark’s final Judge and deduplicated within each model–target campaign; no deduplication is applied across campaigns.
Est. Bounty sums the General Award values below using published MSRC Windows Insider Preview Bounty Program rates. These are benchmark estimates, not MSRC-confirmed awards, payouts, or eligibility determinations.
Benchmark access
The evaluation framework is not publicly distributed due to security, ethical, and licensing considerations.
For research inquiries about running benchmark experiments, please contact hwiwonl2@illinois.edu or jk142@illinois.edu.
Ethical Considerations
Experiments with DeepBinary are intended for security research and should be conducted only in authorized environments. Potentially sensitive findings should be handled through coordinated disclosure with the affected vendor.