Skip to content

fix: installer portability, description limits, and eval honesty - #11

Merged
arpitg1304 merged 2 commits into
mainfrom
fix/skill-loading-and-portability
Aug 12, 2026
Merged

fix: installer portability, description limits, and eval honesty#11
arpitg1304 merged 2 commits into
mainfrom
fix/skill-loading-and-portability

Conversation

@arpitg1304

Copy link
Copy Markdown
Owner

Follows #6 and #9 — this is the remaining work from the same review pass. The frontmatter
name/directory fix and the install.sh traversal fix landed there; this covers what was left.

Skill loading

Two descriptions exceeded the 1024-character frontmatter limit, risking truncation or
rejection by the loader:

Skill Before After
robot-bringup 1181 1015
docker-ros2-development 1078 1003

Trimmed by cutting duplicated trigger keywords — multi-stage builds, udev rules, and
X11 forwarding each appeared twice in their own description. No coverage lost.

Installer portability

install.sh was broken on macOS in two places:

  • list_skills used find -printf, which BSD find does not support.
  • The --skills all path used mapfile, absent from the bash 3.2 that macOS ships.

Both replaced with portable equivalents, verified under real /bin/bash 3.2: --list,
--skills all (10/10 skills), the default bundle, explicit selection, and the exit codes
for missing-target (2) and unknown-skill (1).

Eval honesty

Neither generated package in evals/ actually builds:

  • without-skillsCMakeLists.txt never installs the Python package or the node, so
    nothing is runnable.
  • with-skillsinstall(PROGRAMS ...) lacks RENAME, so the executable is
    demo_recorder_node.py while the launch file requests demo_recorder_node.
  • both — a setup.py that is dead under <build_type>ament_cmake</build_type>.

These are documented rather than patched. They're verbatim agent output, and editing
them would destroy their value as evidence. Added a defects table, a banner on each package
README, methodology caveats (n=1, self-assessed, line counts measure volume rather than
quality), and corrected two conclusion sentences that claimed more than had been verified.

SROS2 examples

permissions.xml used not_after 2026-01-01 — already expired, so copy-pasting it produced
immediately-invalid permissions. Moved to 2035 and documented expiry as a silent-outage
failure mode: under Enforce, participants are rejected at discovery with no topic error
and no crash.

Prevention

  • scripts/validate_skills.py — name/directory match, description length, frontmatter
    validity, README and install.sh consistency. No dependencies beyond Python 3, so
    contributors can run it directly. It reproduces every frontmatter defect found in this
    review, including the one fix: match skill frontmatter name to parent directory #6 fixed.
  • .github/workflows/validate.yml — runs the validator, shellcheck, and an install.sh
    smoke test on Linux and macOS. The rejection case asserts traversing and injecting
    names are refused before any filesystem operation, so fix: validate skill names before use in install.sh #9's fix cannot silently regress.

The validator also warns above a 1000-line-per-skill budget. Four skills exceed it today;
that's left as documented debt rather than an error, so it never blocks a contribution.

Verification

validator:   0 errors, 4 warnings (the line-budget debt)
install.sh:  syntax OK under bash 3.2; all happy paths pass
rejections:  does-not-exist, ../evals, ../../skills, /etc, "skills;rm" all refused
             no traversal side effects on disk

Note for release notes

Anyone who installed before #6 has stale ros1-development/ros2-development directories.
install.sh removes the new name before copying, so a re-install leaves both and the
stale pair still won't load. Worth telling people to delete those two directories.

🤖 Generated with Claude Code

arpitg1304 and others added 2 commits August 11, 2026 21:23
Follows #6 (skill name/directory match) and #9 (install.sh name validation),
which cover the frontmatter names and the traversal fix. This is the
remaining work from the same review.

Skill loading:
- robot-bringup (1181) and docker-ros2-development (1078) had descriptions
  over the 1024-character frontmatter limit, so both were at risk of being
  truncated or rejected by the loader. Trimmed the duplicated trigger
  keywords in each -- "multi-stage builds", "udev rules" and "X11
  forwarding" all appeared twice -- with no coverage lost.

Installer portability (macOS):
- list_skills used `find -printf`, which BSD find does not support.
- The `--skills all` path used `mapfile`, absent from the bash 3.2 that
  macOS ships.
Both defects made install.sh fail on the platform many users run their
agent on. Replaced with portable equivalents and verified under
/bin/bash 3.2: --list, --skills all, the default bundle, explicit
selection, and the exit codes for unknown-target and unknown-skill.

Eval honesty:
- Neither generated package builds. without-skills never installs the
  Python package or node; with-skills installs demo_recorder_node.py
  without RENAME, so its launch file cannot resolve the executable. Both
  are verbatim agent output, so they are documented rather than patched --
  editing them would destroy their value as evidence.
- Added methodology caveats (n=1, self-assessed, line counts are a proxy
  for volume rather than a result) to the report and README, and corrected
  two conclusion sentences that overstated what had been verified.

SROS2 examples:
- permissions.xml used not_after 2026-01-01, already expired, so
  copy-pasting it produced immediately-invalid permissions. Moved to 2035
  and documented expiry as a silent-outage failure mode: under Enforce,
  participants are rejected at discovery with no topic error and no crash.

Prevention:
- scripts/validate_skills.py checks name/directory match, description
  length, frontmatter validity, and README/install.sh consistency. No
  dependencies beyond Python 3. Warns above a 1000-line-per-skill budget.
  It reproduces every frontmatter defect found in this review.
- .github/workflows/validate.yml runs it on each PR, plus shellcheck and
  an install.sh smoke test on Linux and macOS. The rejection case asserts
  that traversing and injecting names are refused before any filesystem
  operation, so the #9 fix cannot silently regress.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shellcheck, newly running in CI, flagged `rm -rf "${target}/${skill}"`:
if target were ever empty this expands to `rm -rf /${skill}`. The
missing-target guard upstream means it cannot happen today, but this is
the one line in the script where being wrong is unrecoverable, so it gets
${target:?} rather than a suppression.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@arpitg1304
arpitg1304 merged commit f9bc546 into main Aug 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant