fix: installer portability, description limits, and eval honesty - #11
Merged
Conversation
Follows #6 (skill name/directory match) and #9 (install.sh name validation), which cover the frontmatter names and the traversal fix. This is the remaining work from the same review. Skill loading: - robot-bringup (1181) and docker-ros2-development (1078) had descriptions over the 1024-character frontmatter limit, so both were at risk of being truncated or rejected by the loader. Trimmed the duplicated trigger keywords in each -- "multi-stage builds", "udev rules" and "X11 forwarding" all appeared twice -- with no coverage lost. Installer portability (macOS): - list_skills used `find -printf`, which BSD find does not support. - The `--skills all` path used `mapfile`, absent from the bash 3.2 that macOS ships. Both defects made install.sh fail on the platform many users run their agent on. Replaced with portable equivalents and verified under /bin/bash 3.2: --list, --skills all, the default bundle, explicit selection, and the exit codes for unknown-target and unknown-skill. Eval honesty: - Neither generated package builds. without-skills never installs the Python package or node; with-skills installs demo_recorder_node.py without RENAME, so its launch file cannot resolve the executable. Both are verbatim agent output, so they are documented rather than patched -- editing them would destroy their value as evidence. - Added methodology caveats (n=1, self-assessed, line counts are a proxy for volume rather than a result) to the report and README, and corrected two conclusion sentences that overstated what had been verified. SROS2 examples: - permissions.xml used not_after 2026-01-01, already expired, so copy-pasting it produced immediately-invalid permissions. Moved to 2035 and documented expiry as a silent-outage failure mode: under Enforce, participants are rejected at discovery with no topic error and no crash. Prevention: - scripts/validate_skills.py checks name/directory match, description length, frontmatter validity, and README/install.sh consistency. No dependencies beyond Python 3. Warns above a 1000-line-per-skill budget. It reproduces every frontmatter defect found in this review. - .github/workflows/validate.yml runs it on each PR, plus shellcheck and an install.sh smoke test on Linux and macOS. The rejection case asserts that traversing and injecting names are refused before any filesystem operation, so the #9 fix cannot silently regress. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shellcheck, newly running in CI, flagged `rm -rf "${target}/${skill}"`:
if target were ever empty this expands to `rm -rf /${skill}`. The
missing-target guard upstream means it cannot happen today, but this is
the one line in the script where being wrong is unrecoverable, so it gets
${target:?} rather than a suppression.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #6 and #9 — this is the remaining work from the same review pass. The frontmatter
name/directory fix and the
install.shtraversal fix landed there; this covers what was left.Skill loading
Two descriptions exceeded the 1024-character frontmatter limit, risking truncation or
rejection by the loader:
robot-bringupdocker-ros2-developmentTrimmed by cutting duplicated trigger keywords —
multi-stage builds,udev rules, andX11 forwardingeach appeared twice in their own description. No coverage lost.Installer portability
install.shwas broken on macOS in two places:list_skillsusedfind -printf, which BSD find does not support.--skills allpath usedmapfile, absent from the bash 3.2 that macOS ships.Both replaced with portable equivalents, verified under real
/bin/bash3.2:--list,--skills all(10/10 skills), the default bundle, explicit selection, and the exit codesfor missing-target (2) and unknown-skill (1).
Eval honesty
Neither generated package in
evals/actually builds:CMakeLists.txtnever installs the Python package or the node, sonothing is runnable.
install(PROGRAMS ...)lacksRENAME, so the executable isdemo_recorder_node.pywhile the launch file requestsdemo_recorder_node.setup.pythat is dead under<build_type>ament_cmake</build_type>.These are documented rather than patched. They're verbatim agent output, and editing
them would destroy their value as evidence. Added a defects table, a banner on each package
README, methodology caveats (n=1, self-assessed, line counts measure volume rather than
quality), and corrected two conclusion sentences that claimed more than had been verified.
SROS2 examples
permissions.xmlusednot_after 2026-01-01— already expired, so copy-pasting it producedimmediately-invalid permissions. Moved to 2035 and documented expiry as a silent-outage
failure mode: under
Enforce, participants are rejected at discovery with no topic errorand no crash.
Prevention
scripts/validate_skills.py— name/directory match, description length, frontmattervalidity, README and
install.shconsistency. No dependencies beyond Python 3, socontributors can run it directly. It reproduces every frontmatter defect found in this
review, including the one fix: match skill frontmatter name to parent directory #6 fixed.
.github/workflows/validate.yml— runs the validator, shellcheck, and aninstall.shsmoke test on Linux and macOS. The rejection case asserts traversing and injecting
names are refused before any filesystem operation, so fix: validate skill names before use in install.sh #9's fix cannot silently regress.
The validator also warns above a 1000-line-per-skill budget. Four skills exceed it today;
that's left as documented debt rather than an error, so it never blocks a contribution.
Verification
Note for release notes
Anyone who installed before #6 has stale
ros1-development/ros2-developmentdirectories.install.shremoves the new name before copying, so a re-install leaves both and thestale pair still won't load. Worth telling people to delete those two directories.
🤖 Generated with Claude Code