Skip to content

[Trophy] Cross-skill inconsistency: ASan memory-limit requirements not cross-referenced between aflpp and libfuzzer #181

Description

@Fr3ya

Bug Link

#181

Skill Used

address-sanitizer, aflpp, libfuzzer

Bug Title

Cross-skill inconsistency : ASan memory-limit requirements not cross-referenced between aflpp and libfuzzer

Additional Context (optional)

Summary

When an agent uses AddressSanitizer with both AFL++ and libFuzzer (a common workflow for multi-fuzzer campaigns), the memory-limit requirements are documented independently in each skill with different flag names and no cross-references. This makes it easy for an agent to compile with -fsanitize=address but forget to disable memory limits for one or both fuzzers, resulting in a silently failed fuzzing campaign.

Details

AddressSanitizer maps virtual address space for shadow memory on 64-bit platforms. This is well-documented in the LLVM ASan docs:

"On 64-bit platforms AddressSanitizer maps (but not reserves) 16+ Terabytes of virtual address space. This means that tools like ulimit may not work as usually expected."

Each skill independently documents this constraint, but with different flags:

Skill What it says Flag
address-sanitizer "ASan maps 16+ TB of virtual address space" (no specific fuzzer flag mentioned)
aflpp (line 464) "Memory limit (-m) is not supported with ASan due to 20TB virtual memory reservation" -m none
libfuzzer (line 568) "Requires ~20TB virtual memory (disable memory limits: -rss_limit_mb=0)" -rss_limit_mb=0

The two flags have different names and different semantics (-m caps virtual memory via RLIMIT_AS; -rss_limit_mb caps resident set size via a watchdog thread), and no skill cross-references the other.

Reproduction

Ask an agent equipped with all three skills to:

"Fuzz this C target with AddressSanitizer enabled, using both libFuzzer and AFL++ in parallel mode."

In our testing, the agent correctly compiled with -fsanitize=address but invoked:

  • libFuzzer without -rss_limit_mb=0
  • AFL++ without -m none

Consequence

In a live environment:

  • libFuzzer: The RSS watchdog thread detects memory usage above the default 2 GB limit and aborts the process within seconds. ASan's 16+ TB virtual reservation far exceeds this.
  • AFL++: RLIMIT_AS prevents ASan from mapping its shadow memory at startup. The instrumented binary fails immediately.

In both cases, the fuzzer exits early or never starts. The campaign produces no crashes, and the operator may conclude the target is free of memory-safety bugs when in fact no meaningful fuzzing occurred.

Context

The bug is a case of specification underspecification: each skill's documentation is individually correct, but the composition of skills creates a gap that is not covered by any single specification.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions