Auditing AI-Generated Code at Scale

Everything else in this guide assumes your codebase fits in an agent’s context window. At some point, it won’t. A 124,000-line codebase generated by AI agents has a problem no single agent can solve: contextual drift — where the agent remembers the immediate function but has no idea how it impacts a module 50 files away.

This guide covers when you need external auditing tools, what categories of tools exist, and how to choose between them.

The Problem: Why Agent Self-Review Breaks Down

AI agents are effective code reviewers for small-to-medium codebases (see Code Review for AI). But they hit fundamental limits:

Context Window Limits

Even the largest context windows can’t hold an entire large codebase at once. When an agent reviews a file, it can’t see the 200 other files that depend on it. It reviews locally and misses globally — the same “local fix, global miss” pattern from the audit findings guide, but at architecture scale.

Contextual Drift in AI-Generated Codebases

When AI agents generate code across many sessions, each session makes local decisions that are internally consistent but may conflict with decisions made in other sessions:

  • Session 1 decides user IDs are UUIDs. Session 47 assumes they’re integers.
  • Session 12 creates a utility function. Session 30 creates a duplicate because it never saw Session 12’s work.
  • Session 5 establishes an error handling pattern. Sessions 20-40 each invent their own.

No single agent can detect this drift because no single agent has seen all the sessions. You need a tool that indexes the entire codebase and reasons across it.

The Scale Threshold

You probably need external tooling when:

  • Codebase exceeds ~50,000 lines — too large for a single agent to hold meaningful context
  • Multiple agents contributed — contextual drift is almost guaranteed
  • The project has been through 20+ sessions — accumulated assumptions compound
  • You can’t explain the full architecture yourself — if you don’t know how all the pieces connect, neither does any single agent
  • You’re preparing for production deployment — the stakes justify the tooling investment

Tool Categories

External auditing tools fall into distinct categories. Most large projects need at least one from each category.

Category 1: Repository-Wide Reasoning (Logic & Architecture)

The problem they solve: “Will this change in the API break the payment flow 10 files away?”

These tools index your entire codebase, build a code graph, and follow data flows across the whole project. They find the bugs that live at the seams between modules — exactly where AI-generated code is weakest.

What to look for:

  • Full repository indexing (not just file-by-file scanning)
  • Multi-hop investigation (follows a function call through 5+ files)
  • Architecture-level reasoning (detects inconsistent patterns across modules)
  • Data flow analysis (traces user input from entry point to database)

When you need this: When your main concern is logic bugs, architectural inconsistencies, or the “hidden” bugs that occur when AI agents make assumptions about shared utilities.

Example tools (as of early 2026):

  • Greptile — Indexes your full codebase and builds a code graph. Uses multi-hop investigation to follow data flows across the project. Strongest for ongoing PR enforcement with codebase-aware rules; can also do one-shot reviews but is optimized for continuous use.
  • CodeRabbit — Has a “full repository review” mode, not just PR diffs. Can be triggered to scan entire codebases in a single pass, making it better suited for one-shot audits.

Category 2: Security Scanning (SAST/DAST)

The problem they solve: “Are there SQL injection vulnerabilities, hardcoded secrets, or insecure dependencies hiding in this codebase?”

Static Application Security Testing (SAST) tools analyze source code for vulnerability patterns. Dynamic Application Security Testing (DAST) tools test running applications. AI-generated code is particularly prone to security shortcuts (see Audit Findings — Root Cause 3).

What to look for:

  • Pattern matching across millions of known vulnerability signatures
  • Auto-fix capabilities (generates PRs with fixes, not just reports)
  • Attack path visualization (shows how a vulnerability can be exploited, not just that it exists)
  • Dependency scanning (checks your package.json/requirements.txt against known CVE databases)
  • Secret detection (finds hardcoded API keys, tokens, passwords)

When you need this: Always, for any project heading to production. This is the non-negotiable category. The Automation & Testing guide covers integrating these into CI/CD.

Example tools (as of early 2026):

  • Snyk (DeepCode AI) — Developer-friendly, provides auto-fix PRs. Scans patterns across millions of open-source projects. Good first choice for teams that want actionable results fast.
  • Checkmarx One — Enterprise-grade. Explains attack paths showing exactly how a vulnerability can be exploited. More thorough but heavier setup.
  • Semgrep — Open-source option. Write custom rules that match your project’s specific invariants (e.g., “every findMany needs a take limit”). Particularly powerful for encoding project-specific rules beyond generic vulnerability patterns. Lightweight and CI/CD-friendly.
  • CodeQL — GitHub’s analysis engine. Free for public repos. Good for custom query-based vulnerability detection.
  • DeepSource — Full repo analysis with auto-fix suggestions. Good for TypeScript codebases. Lower setup friction than SonarQube.

Category 3: Noise Reduction & Triage (Reachability Analysis)

The problem they solve: “We have 500 security alerts. Which ones can actually be triggered in production?”

Large codebases generate enormous numbers of findings from security scanners. Most are false positives or unreachable code paths. Without triage, teams either drown in alerts (and start ignoring them) or spend weeks investigating non-issues.

What to look for:

  • Reachability analysis (determines if vulnerable code can actually be executed)
  • Auto-triage (silences alerts for vulnerabilities that can’t be reached)
  • Prioritization by actual risk, not theoretical severity
  • Integration with existing security scanners (aggregates findings from multiple tools)

When you need this: When your security scanner produces more findings than you can reasonably review — typically after the first full scan of a large AI-generated codebase.

Example tools (as of early 2026):

  • Aikido Security — Auto-triages findings by reachability. Good for solo developers or small teams who need security monitoring without a dedicated security engineer.

Category 4: Code Quality & Maintainability

The problem they solve: “This 124,000-line AI-generated codebase works, but is it maintainable?”

AI agents generate code that passes tests but often creates technical debt: duplicated logic, inconsistent patterns, overly complex functions, poor separation of concerns. These tools measure code health and flag maintainability risks.

What to look for:

  • Code smell detection (long functions, deep nesting, high complexity)
  • Duplication detection (finds copy-pasted code that should be shared utilities)
  • Dependency analysis (detects circular dependencies, tightly coupled modules)
  • Technical debt quantification (estimates effort to fix accumulated issues)
  • Trend tracking (shows whether code quality is improving or degrading over time)

When you need this: When the codebase has grown large enough that you can’t manually verify consistency, or when onboarding new team members who need to navigate AI-generated code.

Example tools (as of early 2026):

  • SonarQube / SonarCloud — The established standard for code quality metrics. Higher setup effort (self-hosted option) but comprehensive analysis.
  • Bito — AI-powered line-by-line review focused on performance bottlenecks and code smells. Good at suggesting refactors for AI-generated spaghetti code.

Category 5: Smart Contract Security (Solidity-Specific)

The problem they solve: “Are there reentrancy bugs, unchecked return values, or economic exploits in my Solidity contracts?”

General-purpose SAST tools don’t understand Solidity’s unique attack surface — reentrancy, storage collisions, gas optimization exploits, proxy patterns, and economic invariant violations. Smart contracts need specialized analyzers that understand the EVM and DeFi-specific vulnerability patterns.

What to look for:

  • Reentrancy detection (cross-function, cross-contract, read-only)
  • Storage layout analysis (especially for upgradeable proxy contracts)
  • Symbolic execution (explores all possible execution paths, not just pattern matching)
  • Economic invariant checking (can a user extract more value than deposited?)
  • Gas optimization analysis

When you need this: Any time you’re deploying smart contracts that handle value. These tools complement — but don’t replace — the Smart Contract Auditing guide’s manual checklist and two-pass workflow.

Automated analyzers (as of early 2026):

  • Slither — The standard first-pass Solidity static analyzer, built by Trail of Bits. Fast, catches common vulnerability patterns (reentrancy, uninitialized storage, unchecked calls). Run this on every commit.
  • Mythril — Symbolic execution engine that finds deeper bugs than pattern matching alone. Slower but more thorough — explores actual execution paths to find exploitable states.
  • Aderyn — Rust-based Solidity analyzer that catches different issues than Slither. Fast execution, good as a complementary second scanner.

Professional audit firms (for pre-mainnet):

Automated tools catch ~60-70% of vulnerabilities (see the Smart Contract Auditing stats). For contracts handling significant value, professional human auditors are essential before mainnet deployment:

  • Trail of Bits — Built Slither. Deep expertise in complex DeFi protocols and novel attack vectors.
  • OpenZeppelin — Maintains the most widely used Solidity library. Audits benefit from deep familiarity with standard patterns.
  • Consensys Diligence — Built Mythril. Strong on formal verification and symbolic analysis.
  • Spearbit — Distributed network of independent security researchers. Often catches issues that single-firm audits miss due to diverse reviewer perspectives.

Recommended smart contract audit stack:

  1. Every commit: Slither + Aderyn in CI (fast, catches low-hanging fruit)
  2. Before testnet: Mythril symbolic execution (slower, finds deeper bugs)
  3. Before mainnet: Professional audit firm + the two-pass agent audit workflow

Decision Framework

Start Here: What Are You Worried About?

Your Concern Start With Category
“Will changes in one module break another?” Repository-wide reasoning Logic & Architecture
“Are there security vulnerabilities?” SAST scanner Security
“Too many security alerts, can’t tell what’s real” Reachability analysis Noise Reduction
“The code works but it’s a mess” Code quality scanner Maintainability
“Smart contracts going to mainnet” Slither + Mythril + professional firm Smart Contract Security
“All of the above” SAST first, then repository-wide reasoning Security + Logic

One-Shot Audit vs. Continuous Enforcement

An important distinction: some tools are designed for ongoing PR enforcement (run on every PR, flag regressions) while others are better for one-shot full-repo audits (scan the entire codebase in a single pass).

Mode Best Tools When to Use
One-shot full audit CodeRabbit (full repo mode), SonarQube, Snyk, Mythril Before releases, after major milestones, initial codebase assessment
Continuous PR enforcement Greptile, Semgrep, CodeQL, Slither, Aderyn Every PR, catches regressions, enforces project-specific rules
Both Snyk, SonarCloud, DeepSource Can run in CI and do periodic full scans

If you’re doing your first audit of a large AI-generated codebase, start with a one-shot tool. Then set up continuous enforcement to prevent new issues.

Build Your Audit Stack

For a large AI-generated codebase heading to production, the recommended stack is:

Layer 1 — Continuous (runs on every PR):

  • SAST scanner in CI/CD (Snyk, Semgrep, or CodeQL)
  • Code quality gate (SonarQube/SonarCloud)
  • Dependency vulnerability scanning (built into most SAST tools)

Layer 2 — Periodic (weekly or before releases):

  • Repository-wide architecture review (Greptile or similar)
  • Full security scan with reachability analysis
  • Technical debt assessment

Layer 3 — Milestone (before major deployments):

  • Manual security audit informed by tool findings
  • Architecture review with expert panel evaluation
  • Performance profiling under realistic load

What External Tools Don’t Replace

External tools are force multipliers, not replacements for the practices in this guide:

  • They don’t replace code review. Tools find patterns; humans judge intent and design.
  • They don’t replace tests. Static analysis finds potential bugs; tests prove actual behavior.
  • They don’t replace invariants. Tools check code; invariants define what “correct” means for your system.
  • They don’t replace understanding your own code. If you can’t explain what a module does, no tool can validate it for you.

Integration with Your AI Workflow

Using Tool Output as Agent Context

External tool findings make excellent input for AI agents. Instead of asking an agent to “find bugs” (which requires it to hold the whole codebase in context), give it specific findings to investigate:

Greptile found that UserService.calculateDiscount() is called from
12 different modules, each passing different assumptions about whether
the price includes tax. Investigate all 12 call sites and determine
which ones are incorrect. Here are the findings: [paste report]

This gives the agent a focused task with clear scope — much more effective than open-ended bug hunting on a large codebase.

Feeding Tool Reports into Audits

When running the two-pass audit workflow or plan-driven development, include external tool reports as input:

Before implementing this feature, review the SonarQube report for the
modules you'll be modifying. Address any existing code smells in the
files you touch. Here's the current report: [paste or link report]

CLAUDE.md Rules for Tool Integration 

## External Tool Integration

Before merging any PR:
- CI must pass with zero critical/high SAST findings
- SonarQube quality gate must pass
- Any new security findings must be triaged (real or false positive)

When modifying existing code:
- Check SonarQube for existing issues in files you touch
- Fix existing issues in files you modify (boy scout rule)
- Don't introduce new code smells above [threshold]

Setting Up Your First Audit

If you’ve never used external auditing tools, start with the lowest-friction option:

Step 1: Security Scan (30 minutes)

Most SAST tools offer free tiers for open-source or small projects. Pick one and run it:

# Example: Snyk (free tier available)
npm install -g snyk
snyk auth
snyk test          # Check dependencies for known vulnerabilities
snyk code test     # Static analysis of your source code

Or if your project is on GitHub, enable CodeQL in your repository settings — it’s free and runs automatically on PRs.

Step 2: Code Quality Baseline (1 hour)

Get a baseline measurement of your codebase health:

# Example: SonarCloud (free for open source)
# Connect your GitHub repo at sonarcloud.io
# Or self-host SonarQube with Docker:
docker run -d --name sonarqube -p 9000:9000 sonarqube:community

Don’t try to fix everything at once. Use the baseline to identify the worst hotspots and prioritize.

Step 3: Architecture Review (2-4 hours)

Run a repository-wide analysis tool or, if you don’t want to adopt a tool yet, use an AI agent with a focused prompt:

I have a [X]-line codebase across [Y] files. I need an architecture audit.
For each major module:
1. What does it depend on?
2. What depends on it?
3. Are there circular dependencies?
4. Are there duplicated utilities that should be consolidated?
5. Are there inconsistent patterns (different error handling styles,
   different ID formats, different API response shapes)?

Start by reading the directory structure, then sample 2-3 files from
each major module to identify patterns.

This won’t be as thorough as a dedicated tool, but it’s a free starting point.

What to Expect: Triaging Your First Scan

When you first run a static analysis tool on a large AI-generated codebase, the results will be overwhelming. A 124,000-line TypeScript + Solidity codebase produced 4,700 findings on its first DeepSource scan. Here’s what that actually looked like and how to make sense of it.

The Raw Numbers Will Scare You

Category Count Action
Anti-pattern 3,600 Mostly noise — any types, non-null assertions, style preferences
Bug Risk 1,000 Skim — most are console.log in browser code and async without await
Performance 95 Quick look — unused imports are real dead code but harmless
Secrets 32 Check immediately — turned out to be placeholder credentials in docs
Security 0 Clean

Of 4,700 findings, exactly 2 were worth clicking into (an infinite loop and a syntax error). Both turned out to be false positives — the “infinite loop” was a standard graceful shutdown pattern (while (running) with signal handlers), and the “syntax error” was ESM export default in a .js config file that the parser didn’t recognize.

The Triage Process

Step 1: Check Secrets first. This is the only category where a finding could mean “credentials are exposed right now.” In the real scan, all 32 were placeholder values like changeme_in_production in documentation files. But verify every one — it only takes a few minutes and a real exposed secret is catastrophic.

Step 2: Check Bug Risk for the rare real bugs. Filter out the expected noise:

  • console.log in browser code — expected if you use console.log in development
  • async without await — common when methods are async for interface consistency
  • “Rules of hooks” in non-React files — false positives from hook-like patterns
  • Array index as key — low risk in dashboard rendering

What’s worth investigating: infinite loops, syntax errors, and anything with only 1 occurrence (rare findings are more likely to be real).

Step 3: Skim Performance. Unused imports are real dead code but harmless. Don’t clean them up in a dedicated pass — just remove them when you’re already editing those files.

Step 4: Ignore Anti-patterns unless they’re Critical severity. The any type (418 occurrences), non-null assertions (1,000+), and style preferences are not bugs. They’re technical debt you can address incrementally, not in a panic sweep.

The Critical Lesson: What Static Analysis Cannot Find

After running the scan and finding nothing alarming, you might think: “124,000 lines of AI-generated code and no major issues?” Be skeptical. Static analysis is a hygiene check, not a security audit. Here’s what these tools fundamentally cannot catch:

Business logic bugs:

  • Does the scoring formula compute correctly with edge-case inputs?
  • Does the distribution math work when there are zero winners?
  • Do financial calculations handle rounding correctly at scale?

Integration bugs:

  • Do off-chain and on-chain systems coordinate correctly?
  • Does the event indexer recover after missing blocks?
  • Do API contracts match between services that were generated in different sessions?

Race conditions under load:

  • Two users staking the same resource simultaneously
  • Two workers processing the same job from the queue
  • Concurrent database writes to the same row

Economic exploits:

  • Can someone game the reputation system through circular endorsements?
  • Can timing manipulation front-run scoring or resolution?
  • Are there arbitrage loops between subsystems?

What actually finds these bugs:

  1. End-to-end test suite against a real database (not mocks)
  2. Fuzz testing on financial services — random inputs to staking, scoring, slashing
  3. Manual adversarial review of the highest-risk services (anything involving funds or state transitions)
  4. Load testing with concurrent users hitting the same resources
  5. Professional security audit for smart contracts before mainnet

The bottom line: A clean static analysis scan means your codebase has good hygiene — no secrets, no obvious anti-patterns, no known vulnerability signatures. It does not mean your business logic is correct, your integrations are sound, or your system is secure under adversarial conditions. Static analysis is Layer 1. It’s necessary but nowhere near sufficient.

Common False Positives in AI-Generated Code

These findings appear on almost every AI-generated codebase and are almost always safe to ignore:

Finding Why It Appears Real Risk
any type usage AI uses any for Prisma results, middleware, test mocks None (but makes TypeScript less useful)
Non-null assertions (!) AI uses ! after queries where it “knows” data exists Low (but hides potential null bugs)
async without await Methods are async for interface consistency None
“Infinite loop” in signal handlers Tools don’t understand process.on closures None — standard shutdown pattern
“Syntax error” in config files Parser doesn’t recognize ESM in .js files None — bundler handles it
Secrets in documentation Example credentials in setup guides None (but verify every single one)
Unused imports AI imports types speculatively or for planned features None (dead code, clean up incrementally)

Back to top

Released into the public domain. Use it however helps you build better software.