IBM Bob 2.0 Hackathon · September 2026

We hid 22 security bugs in two small apps and asked IBM Bob to find them.

It found every one of them, didn't raise a single false alarm, and ignored all seven traps we set for it. The fixes took two tries to get right. This site shows what happened, file by file.

Why we built this

Every AI code review demo we've seen goes the same way: point it at some code, watch it list problems, applaud. But you never find out what it missed, or how many of those problems were real. That's the part that matters if you're deciding whether to let a tool like this near your pull requests.

So we flipped it around. We wrote the bugs first, kept the answer key to ourselves, and graded Bob like an exam.

What we learned from round 1

Bob's fixes were the right ones. Parameterised queries, bcrypt, rate limiting, a proper admin check. But when we tried to apply the patches, 8 of 10 failed because the line numbers in the diff headers were off.

Nobody would have noticed until a developer tried to merge them. So we added a rule to Bob's fix skill: every patch has to pass git apply --check before it counts.

It worked. In round 2 all 12 patches applied as Bob handed them in. Watching the run, you can see several subagents check their patch, find it broken, and rebuild it before reporting back.

How the test works

We write the bugs. Two small Express apps, each with vulnerabilities planted in known places. The second one also has decoys: code that looks risky but is actually fine. The answer key lives in a file Bob is told not to open (and can't see anyway, thanks to .bobignore).
Bob audits. We gave Bob its own Security Auditor mode. It can read the whole project but can only write to the audit/ folder. It splits the files across subagents running in parallel, then writes what it finds to a JSON file.
Bob fixes. A second skill hands each finding to its own subagent, which writes a small patch.
A script marks the work. It compares Bob's findings to the answer key, counts misses and false alarms, and tries to apply every patch.

The two rounds

Round 1 is the easy warm-up. Round 2 is the one we actually cared about: quieter bugs and seven traps.

Who would use this

Any team that's already using AI to write code and now has to decide how much to trust it. Before you let an AI reviewer block a release, you'd want to know its hit rate on your own codebase, not on a vendor's demo.

The findings export as SARIF, so they show up in GitHub's code scanning tab like any other scanner. Each one points to an OWASP ASVS requirement and quotes the exact line, which makes them easy to check by hand.

If you only have a minute

Open and click any item. You'll see the line Bob flagged and the patch it wrote.
shows what Bob caught, and which of the traps it walked past.
has a screenshot of every Bob task and what each one cost.

Findings

Download SARIF

Scores

Worked out by scripts/score_audit.mjs. A finding counts as a hit if it names the right kind of bug in the right file, within three lines of where we planted it. Anything else is a false alarm.

The bugs we planted

The traps (safe code that looks risky)

How it works

Everything Bob needs is in the .bob/ folder: the Security Auditor mode, two skills and a short rules file. The pages on this site just read the files Bob wrote.

The flow

flowchart LR
  classDef bob fill:#edf5ff,stroke:#0f62fe,color:#161616;
  classDef file fill:#fcf4d6,stroke:#b28600,color:#161616;
  classDef ui fill:#ffffff,stroke:#8d8d8d,color:#161616;
  classDef gate fill:#defbe6,stroke:#198038,color:#161616;
  U[Developer
in Bob IDE]:::ui -->|"Security Auditor mode"| A[security-audit
skill]:::bob
  A --> S1[explore subagent
file 1]:::bob
  A --> S2[explore subagent
file 2]:::bob
  A --> S3[explore subagent
file N]:::bob
  S1 & S2 & S3 --> M[merge and
double-check]:::bob
  M --> F[(findings.json)]:::file
  F --> G[generate-fixes
one subagent per finding]:::bob
  G --> D[(fixes/*.diff)]:::file
  D --> V{git apply
--check}:::gate
  F & V --> SC[mark against
answer key]:::gate
  SC --> W[this site + SARIF]:::ui

Custom mode

Bob can read anything, but can only save files matching ^audit/.*\.(json|diff)$. It can't touch the code it's auditing.

Subagents

One read-only subagent per file for the audit, one per finding for the fixes. It keeps each piece of work small, and it's cheaper.

Skills

Written instructions Bob follows every time: what to look for, how to rate severity, and the exact JSON to produce.

Rules

Don't read the answer keys, don't edit the apps. The keys are also hidden from Bob with .bobignore.

.bob/custom_modes.yaml

.bob/skills/security-audit/SKILL.md

.bob/skills/generate-fixes/SKILL.md

Run it yourself

git clone https://github.com/ShermaineYap/IBM-BOB2-HACK && cd IBM-BOB2-HACK
npm install

# In Bob IDE, switch to the Security Auditor mode, then:
#   Run the security-audit skill on ledger_app/, write audit/hard/findings.json
#   Run the generate-fixes skill on audit/hard/findings.json

node scripts/score_audit.mjs audit/hard   # marks Bob's work against the answer key
node scripts/to_sarif.mjs audit/hard      # SARIF file for GitHub code scanning
npm run dev                               # this site

Bob sessions

Every task we ran in Bob IDE for this project, including the one that didn't work out. Click a screenshot to enlarge it. The files are in bob_sessions/.