feat: support hierarchical projection composition and dependency ordering#1824
Conversation
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
github-actions
Bot
assigned
abdallahsamabd,
asaadbalum,
haowu1234,
JaredforReal,
liavweiss,
noalimoy,
rootfs,
samzong,
szedan-rh and
Xunzhuo
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned: 📁
|
✅ Supply Chain Security Report — All Clear
Scanned at |
Xunzhuo
left a comment
Xunzhuo
left a comment
There was a problem hiding this comment.
can you give a brief intro for what is before and what is now?
There was a problem hiding this comment.
Pull request overview
Adds hierarchical composition for projection scores so score inputs can reference other scores and mapping output confidences via type: projection, with dependency-aware evaluation and cycle/undefined-ref validation across config, DSL, CLI, and docs/UI.
Changes:
- Adds projection-score dependency validation (cycle detection / undefined reference checks) in Go + Python validators and DSL validator.
- Updates runtime projection evaluation to compute scores in topological order and to allow reading projection inputs from prior scores or mapping output confidences.
- Updates signal dependency expansion, docs, and dashboard help text to reflect projection-to-projection composition.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| website/docs/tutorials/projection/scores.md | Documents type: projection inputs and hierarchical composition examples. |
| src/vllm-sr/cli/validator.py | Adds Python-side cycle/undefined-ref validation for projection score dependencies. |
| src/semantic-router/pkg/dsl/validator_projection_deps.go | Adds DSL projection-score cycle detection. |
| src/semantic-router/pkg/dsl/validator_conflicts.go | Extends DSL validation to accept projection inputs and validate their fields. |
| src/semantic-router/pkg/config/validator_projection_test.go | Adds unit tests for dependency ordering and projection input validation. |
| src/semantic-router/pkg/config/validator_projection.go | Adds Go config validation for dependency ordering and projection inputs. |
| src/semantic-router/pkg/classification/classifier_signal_eval.go | Expands projection dependency expansion to follow transitive score dependencies. |
| src/semantic-router/pkg/classification/classifier_projections_test.go | Adds runtime tests for score-to-score and confidence consumption + ordering. |
| src/semantic-router/pkg/classification/classifier_projections.go | Evaluates scores topologically and supports reading projection inputs from scores/confidences. |
| dashboard/frontend/src/pages/ConfigPageProjectionsSection.tsx | Updates UI help text for projection score inputs. |
| err = validateProjectionScoreDependencyOrder(cfg.Projections.Scores) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| outputNames, err := validateProjectionMappings(cfg, scoreNames) |
There was a problem hiding this comment.
validateProjectionScoreDependencyOrder is invoked before validateProjectionMappings, but supporting value_source: confidence projection inputs requires knowing the mapping outputs (to resolve output name -> mapping.Source score and validate existence). Consider moving dependency validation after mapping validation (or passing mappings/outputNames into the dependency validator) so confidence-based dependencies can be validated and ordered correctly.
| err = validateProjectionScoreDependencyOrder(cfg.Projections.Scores) | |
| if err != nil { | |
| return err | |
| } | |
| outputNames, err := validateProjectionMappings(cfg, scoreNames) | |
| outputNames, err := validateProjectionMappings(cfg, scoreNames) | |
| if err != nil { | |
| return err | |
| } | |
| err = validateProjectionScoreDependencyOrder(cfg.Projections.Scores) |
| func TestValidateProjectionInputProjectionRef_ValidConfidenceSource(t *testing.T) { | ||
| err := validateProjectionInputProjectionRef("test_score", ProjectionScoreInput{ | ||
| Type: SignalTypeProjection, Name: "other_output", Weight: 0.5, ValueSource: "confidence", | ||
| }) | ||
| if err != nil { |
There was a problem hiding this comment.
There are tests that value_source: confidence is syntactically accepted for projection inputs, but there isn't a test asserting that dependency ordering / undefined-reference validation handles confidence refs correctly (i.e., mapping output name -> mapping.Source score). Adding such a test would help prevent regressions in hierarchical composition.
| func buildProjectionScoreAdj(scores []ProjectionScore) map[string][]string { | ||
| adj := make(map[string][]string, len(scores)) | ||
| for _, score := range scores { | ||
| for _, input := range score.Inputs { | ||
| if strings.EqualFold(input.Type, SignalTypeProjection) { | ||
| adj[score.Name] = append(adj[score.Name], input.Name) | ||
| } |
There was a problem hiding this comment.
buildProjectionScoreAdj treats every type: projection input as a score dependency by appending input.Name. This breaks value_source: confidence projection inputs (where name is a mapping output, not a score) and will cause validateProjectionScoreDependencyOrder to reject valid configs. Consider only adding edges for score-valued projection refs (value_source empty/"score"), and for confidence refs translate the output name to its producing mapping source score (requires building an output->source index from cfg.Projections.Mappings).
| func validateProjectionInputProjectionRef(scoreName string, input ProjectionScoreInput) error { | ||
| if input.Name == "" { | ||
| return fmt.Errorf( | ||
| "routing.projections.scores[%q]: projection input requires a name referencing a declared score", | ||
| scoreName, | ||
| ) | ||
| } |
There was a problem hiding this comment.
validateProjectionInputProjectionRef error text and validation assume the projection input name always references a declared score, but value_source: confidence is documented as referencing a mapping output name. This function should validate the name against the correct namespace depending on value_source (score names vs mapping output names) and produce an error message that reflects that.
| for _, input := range score.Inputs { | ||
| if strings.EqualFold(input.Type, config.ProjectionInputKBMetric) { | ||
| usedSignals[strings.ToLower(config.SignalTypeKB+":"+input.KB)] = true | ||
| continue | ||
| } | ||
| for _, input := range score.Inputs { | ||
| if strings.EqualFold(input.Type, config.ProjectionInputKBMetric) { | ||
| usedSignals[strings.ToLower(config.SignalTypeKB+":"+input.KB)] = true | ||
| continue | ||
| } | ||
| usedSignals[strings.ToLower(input.Type+":"+input.Name)] = true | ||
| if strings.EqualFold(input.Type, config.SignalTypeProjection) { | ||
| c.expandScoreInputs(input.Name, scoreByName, usedSignals, visited) | ||
| continue | ||
| } |
There was a problem hiding this comment.
expandScoreInputs treats any type: projection input as a reference to another score (expandScoreInputs(input.Name, ...)). This is incorrect for value_source: confidence projection inputs where input.Name is a mapping output; as a result, transitive base-signal dependencies for the producing score may not be added to usedSignals, and required signals can be skipped during evaluation. Pass an output->source mapping into expandScoreInputs and, for confidence refs, expand the mapping source score instead of the output name.
| | `inputs[].type` | signal family to read from, or `projection` to reference an earlier score | | ||
| | `inputs[].name` | declared signal name, or projection score/output name | | ||
| | `inputs[].weight` | contribution multiplier; negative weights lower the score | | ||
| | `inputs[].value_source` | `binary`, `confidence`, or `raw` behavior | | ||
| | `inputs[].value_source` | `binary`, `confidence`, `raw`, or `score` (for projection inputs) | |
There was a problem hiding this comment.
The doc now states type: projection can reference mapping output confidences, but the earlier field table still primarily frames inputs[].type/name as signal-family + declared signal name / earlier score. Consider explicitly noting here that when type: projection + value_source: confidence, inputs[].name must match a declared mapping output name (not a score) so readers understand what namespace is being referenced.
| func buildScoreAdjacency(scores []config.ProjectionScore) map[string][]string { | ||
| adj := make(map[string][]string, len(scores)) | ||
| for _, s := range scores { | ||
| for _, inp := range s.Inputs { | ||
| if strings.EqualFold(strings.TrimSpace(inp.Type), config.SignalTypeProjection) { | ||
| adj[s.Name] = append(adj[s.Name], inp.Name) | ||
| } | ||
| } | ||
| results.MatchedProjectionRules = append(results.MatchedProjectionRules, output.Name) | ||
| results.SignalConfidences[signalConfidenceKey(config.SignalTypeProjection, output.Name)] = projectionOutputConfidence(mapping, output, scoreValue) | ||
| } |
There was a problem hiding this comment.
buildScoreAdjacency adds an edge for every type: projection input using inp.Name as though it were always a score name. For value_source: confidence inputs, inp.Name is a mapping output band, so this can inject non-existent nodes into the ordering and even append a zero-value score (byName[name]) with an empty Name. The ordering should ignore/translate confidence refs by mapping output name -> mapping.Source score, and only include actual score dependencies in the graph.
| adj: dict[str, list[str]] = {} | ||
| for score in scores: | ||
| deps = [] | ||
| for inp in getattr(score, "inputs", None) or []: | ||
| if (getattr(inp, "type", "") or "").lower() == "projection": | ||
| dep_name = getattr(inp, "name", None) | ||
| if dep_name: | ||
| if dep_name not in score_names: | ||
| errors.append( | ||
| ValidationError( | ||
| field=f"routing.projections.scores[{score.name}]", | ||
| message=f'projection input references undefined score "{dep_name}"', | ||
| ) | ||
| ) | ||
| deps.append(dep_name) |
There was a problem hiding this comment.
validate_projection_score_dependencies currently treats all type: projection inputs as score-to-score dependencies and errors when the referenced name isn't a score. This will incorrectly reject value_source: confidence projection inputs (which reference mapping outputs), and it also can't validate that a referenced output actually exists. Align this with the Go validator/runtime by: (1) only adding score deps when value_source is empty/"score"; (2) for value_source "confidence", resolve output name -> mapping.Source score and validate the output exists; (3) run cycle detection on the resulting score dependency graph.
| adj: dict[str, list[str]] = {} | |
| for score in scores: | |
| deps = [] | |
| for inp in getattr(score, "inputs", None) or []: | |
| if (getattr(inp, "type", "") or "").lower() == "projection": | |
| dep_name = getattr(inp, "name", None) | |
| if dep_name: | |
| if dep_name not in score_names: | |
| errors.append( | |
| ValidationError( | |
| field=f"routing.projections.scores[{score.name}]", | |
| message=f'projection input references undefined score "{dep_name}"', | |
| ) | |
| ) | |
| deps.append(dep_name) | |
| output_to_source_score: Dict[str, str] = {} | |
| for mapping in getattr(projections, "mappings", None) or []: | |
| source_score = getattr(mapping, "source", None) | |
| if not source_score: | |
| continue | |
| for output in getattr(mapping, "outputs", None) or []: | |
| output_name = getattr(output, "name", None) | |
| if output_name: | |
| output_to_source_score[output_name] = source_score | |
| adj: dict[str, list[str]] = {} | |
| for score in scores: | |
| deps = [] | |
| for inp in getattr(score, "inputs", None) or []: | |
| if (getattr(inp, "type", "") or "").lower() != "projection": | |
| continue | |
| dep_name = getattr(inp, "name", None) | |
| if not dep_name: | |
| continue | |
| value_source = (getattr(inp, "value_source", "") or "score").strip().lower() | |
| if value_source == "score": | |
| if dep_name not in score_names: | |
| errors.append( | |
| ValidationError( | |
| field=f"routing.projections.scores[{score.name}]", | |
| message=f'projection input references undefined score "{dep_name}"', | |
| ) | |
| ) | |
| continue | |
| deps.append(dep_name) | |
| continue | |
| if value_source == "confidence": | |
| source_score = output_to_source_score.get(dep_name) | |
| if not source_score: | |
| errors.append( | |
| ValidationError( | |
| field=f"routing.projections.scores[{score.name}]", | |
| message=f'projection input references undefined output "{dep_name}"', | |
| ) | |
| ) | |
| continue | |
| if source_score not in score_names: | |
| errors.append( | |
| ValidationError( | |
| field=f"routing.projections.scores[{score.name}]", | |
| message=( | |
| f'projection input references output "{dep_name}" ' | |
| f'with undefined source score "{source_score}"' | |
| ), | |
| ) | |
| ) | |
| continue | |
| deps.append(source_score) |
| if !scoreNames[input.SignalName] { | ||
| v.addDiag(DiagWarning, pos, | ||
| fmt.Sprintf("%s: projection input %q references undeclared score", context, input.SignalName), nil) | ||
| } | ||
| switch input.ValueSource { | ||
| case "", "score", "confidence": |
There was a problem hiding this comment.
In checkProjectionScoreInput, projection inputs always warn on !scoreNames[input.SignalName] even when input.ValueSource == "confidence". For confidence refs, SignalName is a mapping output name, so this warning is misleading and will fire for valid configs. Consider branching validation by ValueSource: score refs should require a declared score; confidence refs should validate against declared mapping output names (which may require deferring until after mappings are processed).
| if !scoreNames[input.SignalName] { | |
| v.addDiag(DiagWarning, pos, | |
| fmt.Sprintf("%s: projection input %q references undeclared score", context, input.SignalName), nil) | |
| } | |
| switch input.ValueSource { | |
| case "", "score", "confidence": | |
| switch input.ValueSource { | |
| case "", "score": | |
| if !scoreNames[input.SignalName] { | |
| v.addDiag(DiagWarning, pos, | |
| fmt.Sprintf("%s: projection input %q references undeclared score", context, input.SignalName), nil) | |
| } | |
| case "confidence": | |
| // Confidence references use mapping output names rather than declared score names, | |
| // so they must not be validated against scoreNames here. |
| for _, score := range v.prog.ProjectionScores { | ||
| posMap[score.Name] = score.Pos | ||
| for _, input := range score.Inputs { | ||
| if strings.EqualFold(input.SignalType, config.SignalTypeProjection) { | ||
| adj[score.Name] = append(adj[score.Name], input.SignalName) |
There was a problem hiding this comment.
checkProjectionScoreCycles builds edges using input.SignalName for all type: projection inputs. For value_source: confidence inputs, SignalName is a mapping output, not a score, so the dependency graph is wrong (and cycle detection can miss real cycles or report confusing paths). Build an output->source map from v.prog.ProjectionMappings and translate confidence refs to their producing score before adding edges.
| for _, score := range v.prog.ProjectionScores { | |
| posMap[score.Name] = score.Pos | |
| for _, input := range score.Inputs { | |
| if strings.EqualFold(input.SignalType, config.SignalTypeProjection) { | |
| adj[score.Name] = append(adj[score.Name], input.SignalName) | |
| projectionOutputSources := make(map[string]string, len(v.prog.ProjectionMappings)) | |
| for _, mapping := range v.prog.ProjectionMappings { | |
| projectionOutputSources[mapping.Output] = mapping.Source | |
| } | |
| for _, score := range v.prog.ProjectionScores { | |
| posMap[score.Name] = score.Pos | |
| for _, input := range score.Inputs { | |
| if strings.EqualFold(input.SignalType, config.SignalTypeProjection) { | |
| depName := input.SignalName | |
| if strings.EqualFold(input.ValueSource, config.ValueSourceConfidence) { | |
| if source, ok := projectionOutputSources[input.SignalName]; ok { | |
| depName = source | |
| } | |
| } | |
| adj[score.Name] = append(adj[score.Name], depName) |
asaadbalum
force-pushed
the
feat/issue-1758-hierarchical-projection-composition
branch
2 times, most recently
from
8f65926 to
39c5bf3
Compare
Hey @Xunzhuo, addressed in the description (updates). |
asaadbalum
force-pushed
the
feat/issue-1758-hierarchical-projection-composition
branch
from
39c5bf3 to
00b35e9
Compare
| if vs == "confidence": | ||
| src = output_to_source.get(dep_name) | ||
| if not src: | ||
| errors.append( | ||
| ValidationError( | ||
| field=f"routing.projections.scores[{score.name}]", | ||
| message=f'projection input references undefined mapping output "{dep_name}"', | ||
| ) | ||
| ) | ||
| continue | ||
| deps.append(src) | ||
| else: | ||
| if dep_name not in score_names: | ||
| errors.append( | ||
| ValidationError( | ||
| field=f"routing.projections.scores[{score.name}]", | ||
| message=f'projection input references undefined score "{dep_name}"', | ||
| ) | ||
| ) | ||
| deps.append(dep_name) | ||
| adj[score.name] = deps |
There was a problem hiding this comment.
In validate_projection_score_dependencies, undefined score dependencies are still appended to deps even after emitting an error. This makes the DFS traverse non-existent nodes, which can lead to additional/duplicated cycle diagnostics and hides the fact that the dependency set is invalid. Consider only appending dep_name when it exists in score_names (and likewise validate src is a declared score before appending it for confidence refs).
| if not dep_name: | ||
| continue | ||
| vs = (getattr(inp, "value_source", "") or "").strip().lower() | ||
| if vs == "confidence": | ||
| src = output_to_source.get(dep_name) |
There was a problem hiding this comment.
The CLI schema currently constrains ProjectionScoreInput.value_source to {binary, confidence, raw} (see src/vllm-sr/cli/models.py), so configs that use the documented value_source: "score" for type: projection will fail Pydantic parsing before this validator runs. To fully support the feature, update the model/validation to allow "score" (at least when type == "projection").
| for _, dep := range adj[name] { | ||
| visit(dep) | ||
| } | ||
| ordered = append(ordered, byName[name]) | ||
| } |
There was a problem hiding this comment.
topologicalScoreOrder assumes every dependency name exists in byName. If a projection input references an unknown score (e.g., config constructed programmatically without running config validation), visit(dep) will still run and byName[dep] will append a zero-value ProjectionScore (empty Name), leading to writes under results.ProjectionScores[""]. Consider guarding visit/append with an existence check and skipping (or handling) unknown deps defensively.
| continue | ||
| } | ||
| scoreNames[score.Name] = true | ||
| v.checkProjectionScore(score) | ||
| v.checkProjectionScore(score, scoreNames) | ||
| } | ||
|
|
||
| v.checkProjectionScoreCycles() | ||
|
|
There was a problem hiding this comment.
In checkProjections, scoreNames is populated incrementally and then passed into checkProjectionScore during the same loop. This causes projection inputs that reference scores declared later (which is now explicitly supported via topological ordering) to be flagged as "undeclared" warnings. Consider doing a first pass to collect all score names, then a second pass to validate inputs using the complete set.
…ring (vllm-project#1758) Allow projection score inputs to reference earlier projection scores (value_source: score) or mapping output confidences (value_source: confidence) via type: projection. Scores are evaluated in topological order; dependency cycles are rejected at config validation time. Surfaces updated: Go config validator, Go runtime evaluator, Go signal dependency expansion, Go DSL conflict validator, Python CLI validator, dashboard help text, and documentation. Signed-off-by: abalum <[email protected]> Signed-off-by: asaadbalum <[email protected]>
asaadbalum
force-pushed
the
feat/issue-1758-hierarchical-projection-composition
branch
from
00b35e9 to
a9c2ae6
Compare
Summary
Adds support for hierarchical projection composition, allowing projection score inputs to reference other projection scores or mapping output confidences using
type: projection. Scores are evaluated in topological order with cycle detection and undefined reference rejection at validation time.Changes:
validator_projection.go): AddedvalidateProjectionScoreDependencyOrderwith DFS-based topological sort and cycle detection, plusvalidateProjectionInputProjectionReffortype: projectioninput validation. Confidence refs (value_source: confidence) correctly resolve through mapping outputs to their source score for dependency ordering.classifier_projections.go): Scores evaluate in topological order; addedprojectionInputProjectionValuefor reading fromProjectionScoresorSignalConfidences. Dependency graph builders resolve confidence refs to source scores.classifier_signal_eval.go): RecursiveexpandScoreInputsresolves transitive dependencies for projection-type inputs, including confidence-based refs that resolve through mapping outputs.validator_conflicts.go,validator_projection_deps.go): Cycle detection and projection input validation in DSL layer. Confidence refs validate against mapping output names (not score names) and resolve to source scores for cycle detection.validator.py): Mirrored cycle detection and undefined reference checking with confidence-ref awareness.ConfigPageProjectionsSection.tsx): Updated help text to document both score and confidence value sources for projection inputs.scores.md): Added "Hierarchical Composition" section with examples for both score-to-score and confidence-based references.The DSL parser, compiler, and decompiler already handle
value_sourcegenerically for all input types, so no changes were needed there — only the validation and runtime layers required updates.Test Plan
go test -race)golangci-lintwith.golangci.agent.yml) — zero issues from changed filesBefore and After
Before
Projection composition was shallow — scores could only read base signals (keyword, embedding, etc.), and mappings could only read one score. Layered routing constructs like "difficulty → verification pressure → premium reasoning overlay" required flattening everything into a single oversized score or pushing logic into decision rules.
After
Scores can now reference earlier projection scores (
value_source: score) or mapping output confidences (value_source: confidence). The runtime evaluates scores in topological order, and config validation rejects cycles and undefined references.Closes #1758