Align graph rendering with existing z-score pipeline

[d2r3v]

Summary

This PR removes the custom edge scoring/pruning logic in graphing.py and aligns the graph rendering pipeline with the existing z-score workflow already implemented in the repository.

Previously, graph rendering used a separate IDF-based pruning approach. The repository already computes transition z-scores in zscore_calculation.py and uses them in clustering.py to filter statistically significant transitions. This change reuses that pipeline instead of maintaining a second pruning system.

---

Key Changes

1. Reuse existing z-score pipeline

graphing.py now consumes the z-score outputs produced by zscore_calculation.py.

Edges for avg-session team graphs are filtered using:

abs(z_score) >= z_threshold

Filtering is applied using the existing filter_edges_by_zscore logic from clustering.py.

---

2. Remove custom IDF pruning

Removed the following functions from graphing.py:

• build_edge_idf_map
• score_team_edges
• prune_team_edges_distinctive

This eliminates the duplicate pruning system and simplifies the graphing code.

---

3. Structural repair retained

The structural safeguards remain:

• repair_connectivity – restores minimal bridge edges from the raw graph if filtering disconnects the graph
• fix_orphans – restores a minimal incident edge for isolated nodes

These operate only on the displayed node set and use raw edges as candidate bridges.

---

4. CLI configuration

Added a configurable filtering threshold:

--z-threshold FLOAT

Default:

0.5

This default was chosen because stricter statistical cutoffs (e.g. 1.645) produced overly sparse graphs in this dataset. The threshold remains configurable for stricter filtering if needed.

---

Resulting Pipeline

raw avg-session edges
      ↓
z-score computation (existing step)
      ↓
z-score filtering (abs(z) >= threshold)
      ↓
build filtered graph
      ↓
connectivity repair (raw edges as candidates)
      ↓
orphan repair

This keeps the statistical filtering logic consistent with the rest of the pipeline while ensuring graphs remain structurally interpretable.

---

Verification

Tested using multiple thresholds:

Threshold	Result
0	No filtering (baseline graph)
0.5	Balanced sparsity and readability
1.0+	Graphs often became overly sparse

Connectivity repair ensures filtered graphs remain usable when filtering removes bridge edges.

---

Notes

Cluster graphs may still appear disconnected if the aggregated cluster edge data itself contains multiple components. In those cases the warning reflects the underlying data rather than a rendering error.

[Mahatav]

Looks good.

[AdaraPutri]

hey Dhruv thanks for fixing the pruning methods. Pass 2 and Pass 3 sounds like an interesting and solid approach, especially for higher z-score thresholds where the avg-session graphs get really sparse and it’s easy to end up with disconnected components / floating nodes.

I ran the code across all 22 teams (PR labels) with the recommended setting of --z-threshold 0.5 and only one team (Team 11) got flagged as disjoint + “fixed” by Pass 2 (components: 2->1, +1 bridge). But visually the graph looked the same with vs without --no-preserve-connectivity:

Before:

After:

To double check, I added a debug statement right after the Pass 2/3 block in build_markov_graph() (after final_count = len(keep_set)) to compare what we’re rendering vs what the passes are actually keeping. For Team 11 it showed:

So it looks like Pass 2 is adding the bridge edge into keep_set, but it never gets added into G (and we render using for u, v, data in G.edges(data=True)), which means the “bridge” can’t actually appear in the PNG. That would explain why the logs show +1 bridge but the diagram doesn’t change.

I tried again with a higher threshold (--z-threshold 1.645) and it becomes more obvious: 17 teams were identified as disconnected / got +1 bridge, but again the graphs looked identical with vs without preservation, which I think is the same root issue (edges added to keep_set, but not present in G so they aren’t drawable).

I think this can be fixed pretty cleanly by syncing G with keep_set after Pass 2/3:

• for any (u, v) in keep_set that isn’t already in G, add it into G using the weight from G_original
• then recompute probabilities (at least for the affected source nodes, or just recompute all prob values once after the repair)

That way the preservation passes actually affect what gets rendered.

One more thought: I’m not totally sure we need Pass 3 (orphans) in its current form. In theory it’s meant to reattach isolated nodes, but since all_nodes is derived from set(G.nodes()) (i.e., nodes that survived the filtered edge list), nodes that lose all edges due to pruning usually won’t be present to “fix” in the first place. So it might be doing very little in practice unless we expand the node scope, or we can keep it as a guardrail but it may not fire often.

[aliyahnurdafika]

Great work on this PR! I gained a lot of new insights from these updates. I’ve left a few questions and comments to deepen my understanding. Good job!

[aliyahnurdafika]

Nice documentation update! I have a question, would it help to mention exactly which script is responsible for that overwrite step?

[aliyahnurdafika]

Good work on this validation change! We only require the essential columns (pr_id, event) for the timestamp logic consistency.

[aliyahnurdafika]

I have a question, in here we normalize the event cell for timestamp selection,

events = _parse_event_cell(row.get("event"))
ts = _pick_timestamp(row, events)

but we still write the original raw event value:

"event": row.get("event"),

Is the clean output intentionally preserving the original event representation, or should it write the normalized event value instead? Thanks!

[aliyahnurdafika]

These explanations are really helpful! Not only explaining what is being tested, but also why that behavior is expected.

[aliyahnurdafika]

I think it would be helpful to briefly document why specific z_min values (e.g., 1.0, 999) are used in these tests. Maybe a brief comment at the top of the file could clarify the purpose of these values.