Skip to content

Commit 179ff02

Browse files
NiveditJainclaude
andauthored
feat: first-run install prompt on bare failproofai (#378)
* feat: first-run install prompt on bare `failproofai` invocation PostHog showed only ~10% of npm-installed users ever ran `failproofai policies --install` (470 unique installers → 48 over 90d). The no-args dashboard launch now detects "zero hooks installed across any detected CLI" and offers to run the existing interactive policy-selection inline. Non-TTY falls through to the dashboard with a short hint. - src/hooks/first-run-nudge.ts: opt-out via FAILPROOFAI_NO_FIRST_RUN=1, walks every detected CLI/scope to skip if anything is already set up, emits four PostHog events (first_run_nudge_{shown,accepted,declined, skipped_noninteractive}) so the uplift is measurable. - bin/failproofai.mjs: args.length === 0 guard before launch("start"), try/catch-wrapped so the nudge cannot block the dashboard. - scripts/postinstall.mjs: "Next steps" block for !configured && !registered (the brand-new-user case the existing printHooksWarning doesn't cover). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: document the first-run install prompt + FAILPROOFAI_NO_FIRST_RUN Reflect the new no-args behavior shipped in the previous commit. Quickstart in README and introduction.mdx call out that `failproofai policies --install` is now optional — running bare `failproofai` will offer to do it. Env-vars reference gets a new First-run prompt section for FAILPROOFAI_NO_FIRST_RUN. Chinese mirror (docs/zh/introduction.mdx) and the 14 translated env-vars files are intentionally left for the translation-sync PR pattern (see #371). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: add CHANGELOG entry for first-run prompt docs update Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix: drop conflict markers left in CHANGELOG by rebase onto main The rebase auto-merge for #376's Features entry left literal <<<<<<< markers around the two coexisting bullets. Both entries are valid; they're now side-by-side under 0.0.11-beta.2. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent eb22935 commit 179ff02

9 files changed

Lines changed: 652 additions & 3 deletions

File tree

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@
44

55
### Features
66
- Expand PostHog telemetry coverage to close the 16 server-side and 12 web-UI gaps surfaced by the May audit (#376). Server-side adds `cli_install_success` / `cli_install_failure` / `cli_uninstall_success` / `cli_uninstall_failure` / `cli_list_invoked` / `cli_parse_error` / `cli_unexpected_error` / `hook_dispatch_error` (CLI lifecycle outcomes in `bin/failproofai.mjs`), `hook_stdin_error` / `hook_payload_parse_error` (hook handler input errors in `src/hooks/handler.ts`), `policy_evaluation_error` (builtin policy crashes in `src/hooks/policy-evaluator.ts`, distinct from the existing `custom_hook_error`), `custom_policy_validation_failed` / `custom_hooks_load_error` / `policy_params_validation_warning` / `scope_validation_failed` / `hook_write_failed` / `multi_scope_warning_shown` / `cli_detection_summary` / `beta_policies_installed` (manager / loader / install-prompt internals), and `first_install` / `version_changed` (lifecycle detection in `scripts/postinstall.mjs` via a new `~/.failproofai/last-version` file). Web-UI adds `policies_tab_switched` / `activity_filter_changed` (debounced) / `activity_row_toggled` / `activity_copy_clicked` / `activity_pagination_changed` / `cli_selection_toggled` / `cli_install_remove_submitted` / `cli_reinstall_submitted` / `policy_config_modal_opened` / `policy_config_modal_closed` / `action_error_displayed` / `hooks_install_from_error_clicked` via `usePostHog()` in `app/policies/hooks-client.tsx`. The deny-/instruct-only condition at `handler.ts:344` (allow-path tracking) is intentionally left unchanged. All events go through the existing helpers (`trackHookEvent`, `trackInstallEvent`, `captureClientEvent`) and honor `FAILPROOFAI_TELEMETRY_DISABLED=1`.
7+
- Add a first-run install prompt on bare `failproofai` invocations. PostHog showed only ~10% of npm-installed users ever ran `failproofai policies --install`; the no-args dashboard launch now detects "zero hooks installed across any detected CLI" and offers to run the existing interactive policy-selection inline (covering all of Claude Code, Codex, Copilot, Cursor, OpenCode, Pi, Gemini). Non-TTY contexts (CI, piped invocations) print a short stderr hint and fall through to the dashboard. New `src/hooks/first-run-nudge.ts` module, a guard in `bin/failproofai.mjs` before `launch("start")`, plus four new PostHog events (`first_run_nudge_shown`, `_accepted`, `_declined`, `_skipped_noninteractive`) so the uplift is measurable. Postinstall message extended with a "Next steps" block when the brand-new-user case is detected (`!configured && !registered`). Opt-out via `FAILPROOFAI_NO_FIRST_RUN=1`.
8+
9+
### Docs
10+
- Document the new first-run prompt in the README and `docs/introduction.mdx` quickstart snippets (calling out that `failproofai policies --install` is now optional — running bare `failproofai` will offer to do it), and add a new "First-run prompt" section to `docs/cli/environment-variables.mdx` for `FAILPROOFAI_NO_FIRST_RUN=1`. Chinese mirror and the 14 translated env-vars files left for the translation-sync workflow.
711

812
### Breaking
913
- Remove the undocumented cloud auth + event relay subsystem ahead of a from-scratch redesign. Deletes `src/auth/` (OAuth 2.0 device-flow login against `api.befailproof.ai`, `~/.failproofai/auth.json` token store) and `src/relay/` (WebSocket event relay daemon, sanitized JSONL queue at `~/.failproofai/cache/server-queue/`, PID tracking). Strips the `failproofai login` / `logout` / `whoami` / `relay start|stop|status` / `sync` subcommands and the internal `--relay-daemon` mode from `bin/failproofai.mjs`, along with their `--help` entries and "did you mean" suggestions. Removes the fire-and-forget `appendToServerQueue` + `ensureRelayRunning` calls from `src/hooks/handler.ts` so hook evaluation no longer enqueues events or lazy-spawns a daemon. The whole subsystem had zero references in `README.md`, `docs/`, `examples/`, or `__tests__/`, and only had internal cross-imports — `tsc`, `eslint`, `vitest` (1623 tests), and the `bun run build` bundles all stay green. Users who ran `failproofai login` should also wipe `~/.failproofai/{auth.json,cache/server-queue,relay.pid}` and stop any running relay daemon by hand; new auth/cloud surface will land in a follow-up.

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,11 +81,11 @@ before they become incidents. Zero latency. Runs locally.
8181

8282
```sh
8383
npm install -g failproofai
84-
failproofai policies --install
84+
failproofai policies --install # or just run `failproofai` and accept the first-run prompt
8585
failproofai
8686
```
8787

88-
30 built-in policies activate immediately. Dashboard at `localhost:8020`.
88+
30 built-in policies activate immediately. Dashboard at `localhost:8020`. Disable the first-run prompt with `FAILPROOFAI_NO_FIRST_RUN=1`.
8989

9090
---
9191

Lines changed: 282 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,282 @@
1+
// @vitest-environment node
2+
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
3+
import { PassThrough } from "node:stream";
4+
5+
vi.mock("../../src/hooks/integrations", () => ({
6+
detectInstalledClis: vi.fn(),
7+
getIntegration: vi.fn(),
8+
}));
9+
10+
vi.mock("../../src/hooks/manager", () => ({
11+
installHooks: vi.fn(async () => undefined),
12+
}));
13+
14+
vi.mock("../../src/hooks/hook-telemetry", () => ({
15+
trackHookEvent: vi.fn(async () => undefined),
16+
}));
17+
18+
vi.mock("../../lib/telemetry-id", () => ({
19+
getInstanceId: vi.fn(() => "test-distinct-id"),
20+
}));
21+
22+
function makeIntegration(displayName: string, scopes: readonly string[], installed: boolean) {
23+
return {
24+
id: displayName.toLowerCase(),
25+
displayName,
26+
scopes,
27+
eventTypes: [],
28+
getSettingsPath: vi.fn(),
29+
readSettings: vi.fn(),
30+
writeSettings: vi.fn(),
31+
buildHookEntry: vi.fn(),
32+
isFailproofaiHook: vi.fn(),
33+
writeHookEntries: vi.fn(),
34+
removeHooksFromFile: vi.fn(),
35+
hooksInstalledInSettings: vi.fn(() => installed),
36+
detectInstalled: vi.fn(),
37+
};
38+
}
39+
40+
interface IO {
41+
stdin: PassThrough & { isTTY?: boolean };
42+
stdout: PassThrough & { isTTY?: boolean };
43+
output: string;
44+
}
45+
46+
function makeIO(isTTY: boolean): IO {
47+
const stdin = new PassThrough() as PassThrough & { isTTY?: boolean };
48+
const stdout = new PassThrough() as PassThrough & { isTTY?: boolean };
49+
stdin.isTTY = isTTY;
50+
stdout.isTTY = isTTY;
51+
let output = "";
52+
stdout.on("data", (chunk) => {
53+
output += chunk.toString();
54+
});
55+
const io = { stdin, stdout } as IO;
56+
Object.defineProperty(io, "output", { get: () => output });
57+
return io;
58+
}
59+
60+
async function importModule() {
61+
return await import("../../src/hooks/first-run-nudge");
62+
}
63+
64+
async function importMocks() {
65+
const integrations = await import("../../src/hooks/integrations");
66+
const manager = await import("../../src/hooks/manager");
67+
const telemetry = await import("../../src/hooks/hook-telemetry");
68+
return { integrations, manager, telemetry };
69+
}
70+
71+
describe("hooks/first-run-nudge", () => {
72+
let exitSpy: ReturnType<typeof vi.spyOn>;
73+
74+
beforeEach(() => {
75+
vi.clearAllMocks();
76+
vi.resetModules();
77+
delete process.env.FAILPROOFAI_NO_FIRST_RUN;
78+
exitSpy = vi.spyOn(process, "exit").mockImplementation(((code?: number) => {
79+
throw new Error(`exit:${code ?? 0}`);
80+
}) as never);
81+
});
82+
83+
afterEach(() => {
84+
exitSpy.mockRestore();
85+
});
86+
87+
it("returns immediately when FAILPROOFAI_NO_FIRST_RUN=1 (no detection, no telemetry)", async () => {
88+
process.env.FAILPROOFAI_NO_FIRST_RUN = "1";
89+
const { maybeRunFirstRunNudge } = await importModule();
90+
const { integrations, manager, telemetry } = await importMocks();
91+
92+
await maybeRunFirstRunNudge(makeIO(true));
93+
94+
expect(integrations.detectInstalledClis).not.toHaveBeenCalled();
95+
expect(manager.installHooks).not.toHaveBeenCalled();
96+
expect(telemetry.trackHookEvent).not.toHaveBeenCalled();
97+
expect(exitSpy).not.toHaveBeenCalled();
98+
});
99+
100+
it("returns when no CLIs are detected", async () => {
101+
const { maybeRunFirstRunNudge } = await importModule();
102+
const { integrations, manager, telemetry } = await importMocks();
103+
vi.mocked(integrations.detectInstalledClis).mockReturnValue([]);
104+
105+
await maybeRunFirstRunNudge(makeIO(true));
106+
107+
expect(manager.installHooks).not.toHaveBeenCalled();
108+
expect(telemetry.trackHookEvent).not.toHaveBeenCalled();
109+
expect(exitSpy).not.toHaveBeenCalled();
110+
});
111+
112+
it("returns when any detected CLI already has hooks installed in any scope", async () => {
113+
const { maybeRunFirstRunNudge } = await importModule();
114+
const { integrations, manager, telemetry } = await importMocks();
115+
vi.mocked(integrations.detectInstalledClis).mockReturnValue(["claude", "codex"] as never);
116+
const claudeInt = makeIntegration("Claude Code", ["user", "project", "local"], false);
117+
const codexInt = makeIntegration("Codex", ["user", "project"], true);
118+
vi.mocked(integrations.getIntegration).mockImplementation(
119+
(id: string) => (id === "claude" ? claudeInt : codexInt) as never,
120+
);
121+
122+
await maybeRunFirstRunNudge(makeIO(true));
123+
124+
expect(manager.installHooks).not.toHaveBeenCalled();
125+
expect(telemetry.trackHookEvent).not.toHaveBeenCalled();
126+
});
127+
128+
it("non-TTY: prints hint and fires _skipped_noninteractive", async () => {
129+
const { maybeRunFirstRunNudge } = await importModule();
130+
const { integrations, manager, telemetry } = await importMocks();
131+
vi.mocked(integrations.detectInstalledClis).mockReturnValue(["claude"] as never);
132+
vi.mocked(integrations.getIntegration).mockReturnValue(
133+
makeIntegration("Claude Code", ["user"], false) as never,
134+
);
135+
136+
const io = makeIO(false);
137+
await maybeRunFirstRunNudge(io);
138+
139+
expect(io.output).toContain("No policies are installed");
140+
expect(io.output).toContain("Launching dashboard");
141+
expect(manager.installHooks).not.toHaveBeenCalled();
142+
expect(telemetry.trackHookEvent).toHaveBeenCalledWith(
143+
"test-distinct-id",
144+
"first_run_nudge_skipped_noninteractive",
145+
{ detected_clis: ["claude"], detected_count: 1 },
146+
);
147+
});
148+
149+
it("TTY accept (Y): fires _shown then _accepted, calls installHooks with detected CLIs, exits 0", async () => {
150+
const { maybeRunFirstRunNudge } = await importModule();
151+
const { integrations, manager, telemetry } = await importMocks();
152+
vi.mocked(integrations.detectInstalledClis).mockReturnValue(["claude", "codex"] as never);
153+
const intMap: Record<string, ReturnType<typeof makeIntegration>> = {
154+
claude: makeIntegration("Claude Code", ["user"], false),
155+
codex: makeIntegration("Codex", ["user"], false),
156+
};
157+
vi.mocked(integrations.getIntegration).mockImplementation(
158+
(id: string) => intMap[id] as never,
159+
);
160+
161+
const io = makeIO(true);
162+
setTimeout(() => io.stdin.write("y\n"), 10);
163+
164+
await expect(maybeRunFirstRunNudge(io)).rejects.toThrow("exit:0");
165+
166+
expect(io.output).toContain("Failproof AI — first-run setup");
167+
expect(io.output).toContain("Claude Code, Codex");
168+
169+
const calls = vi.mocked(telemetry.trackHookEvent).mock.calls;
170+
const events = calls.map((c) => c[1]);
171+
expect(events).toEqual(["first_run_nudge_shown", "first_run_nudge_accepted"]);
172+
expect(calls[1][2]).toMatchObject({
173+
detected_clis: ["claude", "codex"],
174+
detected_count: 2,
175+
target_scope: "user",
176+
source: "first-run-nudge",
177+
});
178+
179+
expect(manager.installHooks).toHaveBeenCalledWith(
180+
undefined,
181+
"user",
182+
undefined,
183+
false,
184+
"first-run-nudge",
185+
undefined,
186+
false,
187+
["claude", "codex"],
188+
);
189+
});
190+
191+
it("TTY accept on empty Enter (default Y): runs installHooks", async () => {
192+
const { maybeRunFirstRunNudge } = await importModule();
193+
const { integrations, manager } = await importMocks();
194+
vi.mocked(integrations.detectInstalledClis).mockReturnValue(["claude"] as never);
195+
vi.mocked(integrations.getIntegration).mockReturnValue(
196+
makeIntegration("Claude Code", ["user"], false) as never,
197+
);
198+
199+
const io = makeIO(true);
200+
setTimeout(() => io.stdin.write("\n"), 10);
201+
202+
await expect(maybeRunFirstRunNudge(io)).rejects.toThrow("exit:0");
203+
expect(manager.installHooks).toHaveBeenCalled();
204+
});
205+
206+
it("TTY decline (n): fires _declined with reason user_no, does NOT call installHooks, does NOT exit", async () => {
207+
const { maybeRunFirstRunNudge } = await importModule();
208+
const { integrations, manager, telemetry } = await importMocks();
209+
vi.mocked(integrations.detectInstalledClis).mockReturnValue(["claude"] as never);
210+
vi.mocked(integrations.getIntegration).mockReturnValue(
211+
makeIntegration("Claude Code", ["user"], false) as never,
212+
);
213+
214+
const io = makeIO(true);
215+
setTimeout(() => io.stdin.write("n\n"), 10);
216+
217+
await maybeRunFirstRunNudge(io);
218+
219+
const events = vi.mocked(telemetry.trackHookEvent).mock.calls.map((c) => c[1]);
220+
expect(events).toEqual(["first_run_nudge_shown", "first_run_nudge_declined"]);
221+
const declined = vi
222+
.mocked(telemetry.trackHookEvent)
223+
.mock.calls.find((c) => c[1] === "first_run_nudge_declined")?.[2];
224+
expect(declined).toMatchObject({ reason: "user_no" });
225+
expect(manager.installHooks).not.toHaveBeenCalled();
226+
expect(exitSpy).not.toHaveBeenCalled();
227+
});
228+
229+
it("TTY SIGINT: fires _declined with reason sigint and exits 130", async () => {
230+
// Mock readline so we can trigger the SIGINT handler directly — emulating
231+
// ^C through a PassThrough is brittle across Node versions.
232+
vi.doMock("node:readline", () => ({
233+
createInterface: () => {
234+
const handlers: Record<string, () => void> = {};
235+
return {
236+
on: (ev: string, cb: () => void) => {
237+
handlers[ev] = cb;
238+
},
239+
question: (_q: string, _cb: () => void) => {
240+
setImmediate(() => handlers["SIGINT"]?.());
241+
},
242+
close: () => {},
243+
};
244+
},
245+
}));
246+
247+
const { maybeRunFirstRunNudge } = await importModule();
248+
const { integrations, manager, telemetry } = await importMocks();
249+
vi.mocked(integrations.detectInstalledClis).mockReturnValue(["claude"] as never);
250+
vi.mocked(integrations.getIntegration).mockReturnValue(
251+
makeIntegration("Claude Code", ["user"], false) as never,
252+
);
253+
254+
const io = makeIO(true);
255+
await expect(maybeRunFirstRunNudge(io)).rejects.toThrow("exit:130");
256+
257+
const declined = vi
258+
.mocked(telemetry.trackHookEvent)
259+
.mock.calls.find((c) => c[1] === "first_run_nudge_declined")?.[2];
260+
expect(declined).toMatchObject({ reason: "sigint" });
261+
expect(manager.installHooks).not.toHaveBeenCalled();
262+
vi.doUnmock("node:readline");
263+
});
264+
265+
it("survives a broken integration.hooksInstalledInSettings (treats it as not-installed)", async () => {
266+
const { maybeRunFirstRunNudge } = await importModule();
267+
const { integrations, manager } = await importMocks();
268+
vi.mocked(integrations.detectInstalledClis).mockReturnValue(["claude"] as never);
269+
const broken = makeIntegration("Claude Code", ["user"], false);
270+
broken.hooksInstalledInSettings = vi.fn(() => {
271+
throw new Error("boom");
272+
});
273+
vi.mocked(integrations.getIntegration).mockReturnValue(broken as never);
274+
275+
const io = makeIO(true);
276+
setTimeout(() => io.stdin.write("n\n"), 10);
277+
278+
await maybeRunFirstRunNudge(io);
279+
280+
expect(manager.installHooks).not.toHaveBeenCalled();
281+
});
282+
});

0 commit comments

Comments
 (0)