Skip to content

How It Works

Jump to bottom
PCART Bot edited this page May 8, 2026 · 5 revisions

How PCART Works

This page explains PCART's internal mechanisms: the pipeline, pkl lifecycle, matching strategy, and data flow.

Pipeline Overview

Source Code ──► Preprocess ──► API Extract ──► API Map ──► Compatibility Analyze ──► Auto Repair ──► Report
                  │                │              │              │                      │
                  ▼                ▼              ▼              ▼                      ▼
             Code flattening   AST-based     Dynamic+Static   Change type          AST-level fix
             + instrumentation call discovery  signature match  detection          + dynamic validate

Step-by-step

  1. Preprocess: Flattens control flow (list/dict comprehensions, conditional returns), converts tabs to spaces, merges multi-line calls into single lines, then instruments the code with recordValue.py to capture runtime API call data.
  2. API Extract: Walks the project AST to discover all calls to the target library, resolving aliases from import statements, assignment chains (a = X(); a.f()), and with/async with context managers.
  3. API Map: Matches each discovered API call to its library definition — first via dynamic matching (running the instrumented code), then falling back to static fuzzy matching if dynamic matching fails.
  4. Compatibility Analyze: Compares current vs target API signatures to detect parameter changes (addition, removal, renaming, reordering, type changes, positional↔keyword conversions).
  5. Auto Repair: Applies AST-level fixes for incompatible calls if parameter values can be recovered from pkl files, then validates fixes by running the repaired code in the target environment.
  6. Report: Generates a detailed report in Report/<project>.txt.

pkl File Lifecycle

pkl (pickle) files are the core data artifact that carries runtime API call information between pipeline stages.

Generation (recordValue.py)

When the instrumented project runs in the currentEnv:

  1. paraValueDict collects each API call's receiver object and parameter values at runtime.
  2. apiCoveredSet records which callsites were actually executed.
  3. At process exit (atexit), savePkls() writes pkl files to Copy/pkl/.

Naming

pkl files are named by callsite (callKey), not just the API name. This distinguishes the same API called at different source locations:

<APIName>_<MD5 hash of callsite key>.pkl

For calls within with/async with blocks, two candidate pkl files are generated per callsite:

Suffix Content
__object.pkl Receiver captured as a runtime object via dill
__expr.pkl Receiver captured as a string expression (fallback)

The dual-candidate approach handles cases where the runtime object cannot be pickled (e.g., locked resources, C extension objects).

Manifest

Each callsite also produces a .manifest.json file tracking candidate statuses:

{
  "callsite": "client.publish#_42",
  "covered": true,
  "candidates": [
    {"kind": "object", "status": "saved", "pkl": "client.publish_abc123__object.pkl"},
    {"kind": "expr",  "status": "saved", "pkl": "client.publish_abc123__expr.pkl"}
  ]
}

Candidates marked save_failed are re-generated in the target environment if needed.

Regeneration in Target Environment

If a pkl from currentEnv cannot be loaded in targetEnv (due to serialization format changes), PCART automatically:

  1. Re-instruments the source file for the failed callsite only.
  2. Re-runs the project in targetEnv.
  3. Saves the new pkl with a new_ prefix (e.g., new_client.publish_abc123__object.pkl).

Consumption

  • Dynamic matching (dynamicMatch.py): Loads the pkl, reconstructs the API callable, and uses inspect.signature() to extract the parameter signature.
  • Value addition (addValueForAPI.py): Loads the pkl to fill in concrete parameter values for repair validation.
  • Repair validation (verifySingle.py): Loads the pkl and attempts to call the repaired API with recovered values.

Candidate Order

When multiple pkl files exist for a callsite, PCART tries them in priority order:

  1. new_<key>__object.pkl (re-generated in target env, runtime object)
  2. new_<key>__expr.pkl (re-generated in target env, expression fallback)
  3. new_<key>.pkl (re-generated in target env, legacy format)
  4. <key>__object.pkl (original from current env, runtime object)
  5. <key>__expr.pkl (original from current env, expression fallback)
  6. <key>.pkl (original from current env, legacy format)

Dynamic vs Static Matching

PCART uses a two-tier matching strategy:

Dynamic Matching (preferred)

  1. Load the pkl file for the callsite.
  2. Reconstruct the API callable using the runtime receiver object and parameter values.
  3. Call inspect.signature() to get the actual runtime signature.
  4. Record the internal file path (inspect.getfile()) for disambiguation.

Falls back to static when:

  • No pkl exists for the callsite (not covered by test execution).
  • inspect.signature() returns nullptr (built-in or C extension API).
  • pkl fails to load (serialization error).

Static Matching (fallback)

  1. Match the API call name against the pre-extracted library API definitions (from LibAPIExtraction/).
  2. Use fuzzy matching: match by the last segment of the API name, then filter by name overlap.
  3. Check for import aliases via the library's __init__.py assignments.
  4. Distinguish between .pyi-declared (built-in) APIs and .py-declared APIs.

withitem / async with Support

PCART detects API calls through with and async with context managers:

async with aiofiles.open("file.txt") as f:   # ← withitem: aiofiles.open("file.txt")
    await f.read()                            # ← alias "f" → aiofiles.open("file.txt").read()

How it works:

  1. Extraction (WithVisitor): Records each withitem's context_expr (the API call) and optional_vars (the alias), along with the line number range of the with block.
  2. Alias resolution (modifyWithName): When a call uses the alias (e.g., f.read()), it is recursively resolved to the full API path (e.g., aiofiles.open("file.txt").read()). For nested with blocks with same-named aliases, the innermost scope takes precedence.
  3. Receiver capture (recordValue.py): For withitem callers, both the runtime object and the string expression are saved as separate pkl candidates, since the runtime object may not survive serialization.

callsite-based Identification (callKey)

To distinguish the same API called at different source locations, PCART uses a composite key:

callKey = "<API_name>#_<line_number>"

This key is used consistently throughout the pipeline:

  • Preprocess: As the dictionary key in paraValueDict instrumentation.
  • pkl naming: As the base for pkl file names.
  • Dynamic matching: As the lookup key to find the correct pkl.
  • Shared dictionary: As the cache key to avoid redundant matching across files.

Without callsite keys, two calls to the same API at different lines would share one pkl and one match result, causing incorrect signature assignments.

Temporary Directories

Directory Purpose Lifecycle
Copy/<projName>/ Instrumented project copy for pkl generation Recreated each run
Copy/bak_<projName>/ Backup of instrumented copy for single-API re-instrumentation Recreated each run
Copy/pkl/ Runtime pkl files and manifests Recreated each run
Dynamic/<projName>/ Stripped-down project copy for dynamic matching scripts Recreated each run
data/ Intermediate JSON files (match snapshots, call dicts) Recreated each run

Related Pages

Clone this wiki locally