ci: pin validator deps, check SKILL.md frontmatter, catch unlisted plugin dirs, run script-package tests

[deanrie]

Summary

One PR for the repo-level CI and validator work from #531, replacing #607, #599, #598 and #625 (closed in favour of this one; their commits are carried here unchanged, one commit each, reviewable separately). Together they close pieces 2, 3b, 3d and 4 of the shape in #531, the last ones; #582 (akushonkamen) covers pieces 1 and 2a and touches validate-plugins.yml / validate-plugins.mjs too, so whichever lands second needs a small rebase.

1. ci: pin the validator's ajv dependencies in a root package.json and install with npm ci (was ci: pin the validator's ajv dependencies in a root package.json and install with npm ci #607). The job installed ajv and ajv-formats unpinned on every run. Root package.json (private, devDependencies only, validate script), package-lock.json (7 packages), npm ci with the npm cache, and the paths: trigger widened to scripts/**, the package files and the workflow.

2. validate: check SKILL.md frontmatter (name, description, folder match, length) (was validate: check SKILL.md frontmatter (name, description, folder match, length) #599). The validator never read a SKILL.md. Errors: no frontmatter, missing name, missing description. Warnings for now: name not kebab-case or not equal to the folder, description over 1024 characters. On current main that is 0 errors and 5 warnings (cursor-sdk description, two pstack and two third_party names), all with open fix PRs (chore: keep the cursor-sdk skill description under the 1024-char limit; label agent-compatibility changelog 1.0.0 #530, fix(pstack): make skill names match their directories (poteto-mode, make-bot-ui) #596, fix(x, x-money): make skill names match their directories #600); once those land the two warnings can become errors in a two-line follow-up.

3. ci: run bun typecheck and tests for the two script packages on PRs (was ci: run bun typecheck and tests for the two script packages on PRs #598). orchestrate/skills/orchestrate/scripts (209 tests) and pstack/skills/poteto-mode/scripts (63 tests) have test suites nothing ran. New script-packages.yml: matrix over the two directories, bun install --frozen-lockfile, bun run typecheck, bun run test, fail-fast: false, paths: scoped to the two packages and the workflow.

4. validate: fail when a plugin directory is not listed in marketplace.json (was validate: fail when a plugin directory is not listed in marketplace.json #625). The validator only walked marketplace.json entries, so a directory with a .cursor-plugin/plugin.json that no entry references (forgotten listing, or a source typo) passed silently. New step walks the repo root and third_party/ and fails for each plugin directory that is not a listed source. On current main: 102 listed, 0 unlisted, so this is an error from day one.

Verification

• rm -rf node_modules && npm ci && npm run validate on this branch: All plugins validated successfully. with the 5 expected warnings.
• Planted zz-unlisted/.cursor-plugin/plugin.json: ERROR: Unlisted plugin: "zz-unlisted" ..., exit 1.
• Both workflow files parse as YAML.
• Script packages checked locally with bun 1.4.3: orchestrate tsc ok, 209 pass / 0 fail; poteto-mode tsc ok, 63 pass / 0 fail.
• As a fork PR the workflow runs will sit in "approval required" until a maintainer approves them.

Refs #531.

[rahul05ranjan]

Verified most of this locally (standalone run of the new validator logic against the full tree — I didn't have bun/ajv installed, so the script-package matrix I reviewed statically only). Results:

• Copied the new checkSkillFrontmatter/checkPluginSkills code verbatim and ran it over all 102 marketplace plugins: 0 errors, 5 warnings — exactly the five you listed (cursor-sdk's 1045-char description, make-bot-ui/poteto-mode/x-api-mcp-guide/x-money-guide names). Matches the PR description to the letter.
• The unlisted-plugin walk reports 102 listed / 0 unlisted on main, and I planted a zz-unlisted/.cursor-plugin/plugin.json to confirm it gets caught with exit-1 semantics. Also confirmed every current marketplace source is a root-level dir or third_party/*, so the two-level walk covers everything that exists today.
• Frontmatter parser edge cases behave: quoted scalars strip quotes, >- folds lines, | preserves blank lines, missing/unterminated frontmatter returns null, and the folder-mismatch warning fires when a kebab-case name differs from its directory.
• Both workflow files parse as YAML, and the package.json validate script wiring matches the workflow step. Lockfile is exactly the 7 claimed entries.

Two small nits, neither blocking:

1. parseFrontmatter (scripts/validate-plugins.mjs:53): text.indexOf("\n---", 3) also matches a continuation line that merely starts with ---, so a block-scalar description containing e.g. a markdown rule ( --- text) silently truncates the value at that point. Anchoring the terminator to a line that is exactly --- would close that; as-is it's a silent mis-parse rather than an error, which is the one spot this new checking can be quietly wrong.

2. script-packages.yml:33 pins bun to latest — a little at odds with this PR's own rationale for pinning the ajv deps in CI. Not urgent, but a future bun release could start failing these checks for reasons unrelated to any PR.

Thanks for folding the four pieces into one reviewable-per-commit PR — the body made it genuinely easy to verify each claim independently.

[deanrie]

Hey @rahul05ranjan, thanks for the thorough pass, and for replicating the walk against all 102 plugins. Both nits addressed in ba400c7:

1. The frontmatter terminator is now anchored to a line that is exactly --- (optional trailing whitespace, CRLF or EOF after it), so an indented --- text continuation inside a block scalar stays part of the value. Checked: a | description containing --- not a terminator round-trips intact, ---- is not accepted as a terminator, CRLF files and a terminator at EOF still parse. The full tree still yields 0 errors and the same 5 warnings.
2. bun-version pinned to 1.4.3 (current release) in script-packages.yml, same reasoning as the ajv pins.