Skip to content

chore: add analytics to chat#1331

Open
leonardmq wants to merge 2 commits intomainfrom
leonard/kil-601-posthog-analytics-for-chat
Open

chore: add analytics to chat#1331
leonardmq wants to merge 2 commits intomainfrom
leonard/kil-601-posthog-analytics-for-chat

Conversation

@leonardmq
Copy link
Copy Markdown
Collaborator

@leonardmq leonardmq commented Apr 23, 2026

What does this PR do?

Add analytics events to chat.

New events:

Event Fields
chat_message_sent is_new_conversation (boolean), message_length (number), has_app_context_header (boolean), message_count (number)
chat_conversation_started
chat_stopped
chat_retry
chat_tool_approval_run tool_name (string | undefined)
chat_tool_approval_skip tool_name (string | undefined)
chat_new_chat_clicked had_messages (boolean)
chat_cost_disclaimer_shown
chat_cost_disclaimer_accepted
chat_cost_disclaimer_declined
chat_history_opened
chat_history_session_loaded message_count (number)
chat_history_session_deleted

Checklists

  • Tests have been run locally and passed
  • New tests have been added to any work in /lib

Summary by CodeRabbit

  • Chores
    • Added analytics event tracking across chat flows: message submissions, conversation start/stop/retry, tool approval run/skip, new-chat clicks (with had_messages), cost-disclaimer shown/accepted/declined, and chat-history opened/session loaded/deleted to improve product insights.
  • Bug Fixes
    • Prevented duplicate dismissal of the cost-disclaimer dialog by adding an early return when no pending resolver exists.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 23, 2026

📝 Walkthrough

Walkthrough

Integrates PostHog analytics across chat features: session store emits events for message send/start/stop/retry and tool approvals (resolving tool names); chat UI emits events for new chats, cost-disclaimer lifecycle, and history open/load/delete. Adds a retry flag to streaming start and a guard in disclaimer dismiss.

Changes

Cohort / File(s) Summary
Chat Session Store
app/web_ui/src/lib/chat/chat_session_store.ts
Adds PostHog event captures for chat_message_sent, chat_conversation_started, chat_stopped, chat_retry, chat_tool_approval_run, and chat_tool_approval_skip; beginStreaming accepts isRetry; retry flows route through beginStreaming(..., true); introduces helper to resolve tool_name from toolApprovalWaiter payload.
Chat New / Cost Disclaimer / History UI
app/web_ui/src/routes/(app)/chat/chat.svelte, app/web_ui/src/routes/(app)/chat/chat_cost_disclaimer.svelte, app/web_ui/src/routes/(app)/chat/chat_history.svelte
Emits chat_new_chat_clicked (with had_messages), chat_cost_disclaimer_shown/accepted/declined (adds guard in dismiss() to return when no pending resolver), and chat_history_opened, chat_history_session_loaded (includes message_count), chat_history_session_deleted. Minor guard added to open() in history.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • scosman

Poem

🐇 I hopped through code with nimble paws,

I marked each chat, each pause and cause,
Events I scattered, soft and quick,
From start to retry, every tick,
Analytics carrots—one, two, toss! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'chore: add analytics to chat' is a brief, clear summary of the main change (adding analytics to chat functionality) and accurately reflects the primary purpose of the changeset.
Description check ✅ Passed The description includes most required sections: clear explanation of what the PR does, a detailed table of new events with their fields, and completed checklists confirming tests were run and added. The 'Related Issues' section is missing.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch leonard/kil-601-posthog-analytics-for-chat

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates PostHog analytics across the chat interface, tracking events such as message submission, conversation starts, tool approvals, and history interactions. A logic issue was identified where the chat_conversation_started event could be triggered multiple times if a user retries the first message of a conversation before a trace ID is assigned.

Comment thread app/web_ui/src/lib/chat/chat_session_store.ts
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@app/web_ui/src/lib/chat/chat_session_store.ts`:
- Around line 169-177: beginStreaming currently emits analytics events but is
also called by retryLastRequest, causing retries to be logged as new sends;
update beginStreaming to accept a source parameter (e.g., source: "send" |
"retry"), have sendMessage call beginStreaming("send") and retryLastRequest call
beginStreaming("retry"), then only emit chat_conversation_started when source
=== "send" && !traceId; optionally only emit chat_message_sent when source ===
"send" to keep retries separate from sends. Ensure you update signatures and
callers for beginStreaming and adjust the logic that emits posthog.capture for
chat_message_sent and chat_conversation_started accordingly.

In `@app/web_ui/src/routes/`(app)/chat/chat_history.svelte:
- Around line 51-55: The analytics event is fired before verifying the dialog
actually opened; update the open() function so
posthog.capture("chat_history_opened") is called only after a successful
historyDialog?.show() (i.e., check historyDialog exists and call show() first),
and then trigger posthog.capture; keep void loadSessionList() as-is after
showing. Reference: modify open(), historyDialog?.show(), posthog.capture, and
loadSessionList() so the capture happens post-show and only when historyDialog
is present.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d818490d-2773-4986-8324-dbb330723bb7

📥 Commits

Reviewing files that changed from the base of the PR and between 41ee2ca and 249fb2f.

📒 Files selected for processing (4)
  • app/web_ui/src/lib/chat/chat_session_store.ts
  • app/web_ui/src/routes/(app)/chat/chat.svelte
  • app/web_ui/src/routes/(app)/chat/chat_cost_disclaimer.svelte
  • app/web_ui/src/routes/(app)/chat/chat_history.svelte

Comment thread app/web_ui/src/lib/chat/chat_session_store.ts
Comment thread app/web_ui/src/routes/(app)/chat/chat_history.svelte
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 23, 2026

📊 Coverage Report

Overall Coverage: 92%

Diff: origin/main...HEAD

No lines with coverage information in this diff.


Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
app/web_ui/src/lib/chat/chat_session_store.ts (1)

406-434: toolNameForCallId resolution is correctly ordered before state mutation.

Calling toolNameForCallId(toolCallId) before the combined.update(...) / maybeFinishToolApproval() ensures the lookup still sees toolApprovalWaiter populated — important since maybeFinishToolApproval can clear it. Good sequencing.

One small nit: applyToolApprovalRun and applyToolApprovalSkip are now near-identical aside from the event name and the boolean pick. If you ever add a third decision type, extracting a shared helper (e.g., applyToolApprovalDecision(id, decision)) would cut the duplication, but it's not worth it at two call sites.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@app/web_ui/src/lib/chat/chat_session_store.ts` around lines 406 - 434, Both
applyToolApprovalRun and applyToolApprovalSkip duplicate the same logic; extract
a helper applyToolApprovalDecision(toolCallId: string, pick: boolean, eventName:
string) that first checks get(combined).toolApprovalWaiter, captures the event
via posthog.capture(eventName, { tool_name: toolNameForCallId(toolCallId) }),
performs combined.update to set toolApprovalPicks[toolCallId] = pick, and then
calls maybeFinishToolApproval(); then implement applyToolApprovalRun and
applyToolApprovalSkip as thin wrappers that call
applyToolApprovalDecision(toolCallId, true, "chat_tool_approval_run") and
applyToolApprovalDecision(toolCallId, false, "chat_tool_approval_skip"),
preserving the current ordering (call toolNameForCallId before the state
update).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@app/web_ui/src/lib/chat/chat_session_store.ts`:
- Around line 406-434: Both applyToolApprovalRun and applyToolApprovalSkip
duplicate the same logic; extract a helper applyToolApprovalDecision(toolCallId:
string, pick: boolean, eventName: string) that first checks
get(combined).toolApprovalWaiter, captures the event via
posthog.capture(eventName, { tool_name: toolNameForCallId(toolCallId) }),
performs combined.update to set toolApprovalPicks[toolCallId] = pick, and then
calls maybeFinishToolApproval(); then implement applyToolApprovalRun and
applyToolApprovalSkip as thin wrappers that call
applyToolApprovalDecision(toolCallId, true, "chat_tool_approval_run") and
applyToolApprovalDecision(toolCallId, false, "chat_tool_approval_skip"),
preserving the current ordering (call toolNameForCallId before the state
update).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 555acdc7-6970-45f6-8ad5-a1c7de4e6a01

📥 Commits

Reviewing files that changed from the base of the PR and between 249fb2f and e338688.

📒 Files selected for processing (2)
  • app/web_ui/src/lib/chat/chat_session_store.ts
  • app/web_ui/src/routes/(app)/chat/chat_history.svelte
🚧 Files skipped from review as they are similar to previous changes (1)
  • app/web_ui/src/routes/(app)/chat/chat_history.svelte

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant