AI Safety & Security

Testing what AI systems actually do

Independent research on evaluating coding agents, competitive cybersecurity, and hands-on work building model evaluations.

Research

🔬
Independent Research · Oct 2026
What the Tests Don't See

A static audit of 6,000 agent-generated patches on SWE-bench Verified

Coding agents are graded on whether their patches pass tests, but tests can't see a security weakness slipped in alongside a fix, an error silenced instead of fixed, or a test rewritten to agree with the code. I audited every patch 12 frontier models submitted to SWE-bench Verified under the same agent scaffold, against the human-written fixes for the same issues, using before/after static analysis (Bandit, Semgrep) and behavioral checks.

Result: agent patches are not less secure than human fixes. What the tests miss is behavior: one model routinely adds silent except: pass handlers, and three models edit tests they were explicitly told not to touch, sometimes rewriting assertions to match their own change. SWE-bench can't see either, by design.

AI Safety AI Security LLM Evaluation Python Semgrep · Bandit Statistics
Read the paper Code & data Project page
// 12 models · 6,000 patches · 1 scaffold
 
new_security_finding
  models 1.1% · human 1.6%
  task-intrinsic 85%
 
silent_except
  GPT-5 mini 4.5% · others ≤1.0%
 
forbidden_test_edits 23
  rewrote assertions: 5
 
// test special-casing found: 0

Competitions

🚩
Cybersecurity · June 2026
US Cyber Challenge — East Coast Cyber Camp CTF

Qualified through the US Cyber Challenge's Cyber Quests Spring 2026 online competition for an invitation to the USCC East Coast Cyber Camp. In the camp's capture-the-flag competition I played as dr10 on team SSH_ITS@DARK.

11th
team placement (9,435 points)
4,775 pts
team's top scorer: about half of the team's total
12th
individual placement on the users scoreboard
Final scoreboard page for team SSH_ITS@DARK in the USCC East Coast CTF 2026: 11th place, 9,435 points. Members: Skippy (captain) 4,250; dr10 4,775; смит 360; minshaal 50. A banner reads 'Scoreboard has been frozen'.
Team page from the final, frozen scoreboard, captured June 12, 2026. The competition site is taken offline after each event, so this screenshot is the record of the result.
CTF Capture the Flag US Cyber Challenge

AI Evaluation Experience

✨
Professional Work
Project Helix — Handshake AI

AI training data engineer on a SWE-bench-style model evaluation pipeline. I wrote deterministic fail-to-pass test suites across real open-source codebases in Python, PHP, and Rust, covering security, protocol compliance, and database correctness, each packaged with a Docker-based evaluation runner for automated validation.

SWE-bench pytest / PHPUnit Docker Model Evaluation
More project details