1_refactoring_genetic_algorithms - #2547
Conversation
|
Job Test CentOS 7 on 201ffb0 : invalidated by @Jimmy-INL |
…/--create on Linux. Writes an activation script to set LD_LIBRARY_PATH after env activation.
Strict-review item 1: crowding distance is an objective-space diversity measure, not a fitness measure, so its public parameter is renamed frontUtils.crowdingDistance(..., fitness=) -> objectiveValues= to make the objective/fitness separation explicit at the call boundary. Internal locals (fMax/fMin -> objMax/objMin) and the docstring are updated to match, and the stale "To be removed"/"FIXED" commented block is deleted. All callers updated: NSGAII._process_generation (combined and first-gen paths), the duplicated multi-objective branch in GeneticAlgorithm, and the testFrontUtils unit test. Behavior is unchanged (testFrontUtils 6/6 pass); this is a naming/clarity change only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strict-review "Low" finding: GeneticAlgorithm carried a duplicate, unvalidated hypervolume implementation (_checkConvHypervolume, _computeHypervolume, _hypervolume2D, _hypervolume3D, _hypervolumeWFG). This code was dead in the single-objective base class -- 'hypervolume' is not a registered convergence option for GeneticAlgorithm (objective/AHDp/AHD/HDSM) nor for MultiObjectiveGeneticAlgorithm (which adds spread/spacing/maxSpread/rank1Ratio) -- so the dispatch getattr(self, '_checkConvHypervolume') was never reached. The active multi-objective path lives in MultiObjectiveGeneticAlgorithm, which retains its own _checkConvHypervolume guard (raises NotImplementedError) until a mathematically validated indicator is added. Removing the GA copy eliminates the duplication and the negative/zero-objective reference-point bug it carried. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mechanical, behavior-preserving cleanup addressing Congjian Wang's inline review comments on PR idaholab#2547. No logic or numerical behavior changes. - camelBack convention applied to function-local variables across the GA convergence/diversity helpers (spread, spacing, maxSpread, rank1Ratio) and the generational-distance metrics (_GD, _GDp, _popDist, _ahd, _envelopeSize, _hdsm) in GeneticAlgorithm; the spread/spacing/maxSpread and key-matching helpers in MultiObjectiveGeneticAlgorithm; AutoARMA; and UnorderedCSVDiffer. - Docstrings expanded/fixed: NSGA-II class/methods, MOGA __init__ self-variable docs and metric methods; removed a duplicated/stacked docstring on _envelopeSize and stray "FIXED" annotations. - Dead code removed: stray "# @Profile" in survivorSelection, a debug raiseADebug in MOGA, a commented-out line in testFitnessBased. - dependencies.xml: documented why statsforecast/utilsforecast are pinned (AutoARMA auto_arima test compatibility / conda env resolution). Scope limited to mechanical changes; architectural suggestions (separate Pareto module, vectorization, 2-D population storage, reducing supported input types) are catalogued for follow-up. Optimizers suite: 85 passed, 15 skipped, 0 failed (unchanged from baseline; no gold diffs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strict-review item 7 follow-up. The earlier round disabled hypervolume convergence because the old indicator was unvalidated (multiplicative reference nadir*1.1 invalid for zero/negative objectives; incorrect 2D/3D/WFG formulas; mixed current-minimization-space and previously-exported objective values). This restores it with a correct, testable implementation. frontUtils: - New standalone, unit-testable hypervolume(points, reference) plus the recursive _hypervolumeHSO helper, computed in minimization space with the exact Hypervolume-by-Slicing-Objectives recursion (While et al., 2006), correct for any number of objectives. Points that do not dominate the reference contribute nothing. MultiObjectiveGeneticAlgorithm: - _checkConvHypervolume rebuilt: builds the rank-1 front in internal minimization space (no external/internal sign mixing), keeps a per-trajectory front history, and compares current vs previous hypervolume against a COMMON reference so the relative change is meaningful. - New _hypervolumeReference uses the union nadir plus an ADDITIVE margin (marginFraction * range, floored away from zero), which stays valid for zero or negative objectives, unlike the old multiplicative reference. - 'hypervolume' re-added to convergenceOptions. Tests: testFrontUtils gains exact-value hypervolume checks (2D unit square=1, 3-point front=6, 2-point front=5, 3D unit cube=1, negative objectives=1, dominated point ignored). Full unit suite passes 12/12. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the canonical NSGA-II continuous variation pair for real-valued decision variables, so the multi-objective GA is competitive on continuous (ZDT/DTLZ/engineering) problems where the existing gene-swapping operators are inappropriate. - crossovers.py: sbxCrossover (Simulated Binary Crossover, Deb & Agrawal 1995), distribution index eta; bound-aware via a shared _finiteGeneBounds helper (distribution bounds -> ppf quantiles -> padded observed range). - mutators.py: polynomialMutator (Deb & Goyal 1996), distribution index eta; same bound resolution. - GeneticAlgorithm.py: register both in the crossover/mutation input enums with manual descriptions, and pass distDict into the crossover call so SBX can obtain variable bounds. - Unit tests: testSBXCrossover (6/6) and testPolynomialMutator (7/7), registered in the Optimizers unit-test driver. - Docs: generateOptimizerDoc.py now provides example decks for the MultiObjectiveGeneticAlgorithm and NSGA-II optimizer types (previously the generator crashed with KeyError on these newly-registered types), and regenerated optimizer.tex to include the new operators in all GA enum sections plus the two new optimizer examples. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror NSGAII.getInputSpecification by setting specs.name explicitly on MultiObjectiveGeneticAlgorithm, and correct the docstring (it previously declared "@ Out, None." while the method returns the input spec). Cosmetic only: the generated optimizer.tex is byte-identical before and after this change (the base spec already resolved to the class name), so no documentation regeneration is required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the O(N^3) recursive peeling ranker in the constrained path with the canonical O(M*N^2) fast non-dominated sort (Deb et al., 2002): one pairwise pass records, per individual, the set it dominates and the count that dominate it, then fronts are peeled by decrementing domination counters. Uses the same Deb constrained-dominance relation, so ranks are identical to the previous algorithm; only the cost changes (large speedup at population sizes >= 100). Behavior-preserving, verified in testFrontUtils.py: - known unconstrained (3 fronts) and constrained (feasible-dominates- infeasible) cases; - the public rankNonDominatedFrontiers on the constrained problem; - 25 randomized trials (2-3 objectives, random constraint violations) asserting the new sort matches an independent brute-force reference. Unit suite: 40/40 (was 12/12). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nomial) Wire a continuous multi-objective regression test that actually runs on the NSGA-II algorithm branch. The existing NSGA-II_ZDT1 et al. framework tests are skipped here because their decks reference NSGA-specific OutStream plot types (NSGAParetoFrontPlot, NSGAFrontAnimation, NSGARankHistoryPlot, NSGACrowdingDistancePlot, NSGAFrontRankAnimation) that live on the separate NSGA plotting PR; on this branch they cannot run regardless of golds. ZDT1_sbx.xml is a self-contained variant of the ZDT1 benchmark that uses only standard OutStreams (Print, OptPath, PopulationPlot) and solves it with the real-coded operators (sbxCrossover + polynomialMutator). It therefore: - exercises the new SBX crossover and polynomial mutation end-to-end through the full optimizer path (not just the operator unit tests); - gates the multi-objective ranking / crowding machinery on a continuous problem via an UnorderedCSV diff of the exported Pareto front. The run is deterministic (fixed seeds): re-running produces a byte-identical opt_export_0.csv, which is committed as the gold. Test passes (1/1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reusable diversity/archive primitives for NSGA-II, fully unit-tested and backward compatible (no existing call site changes behavior): - updateParetoArchive(archive, new, minMask, maxArchiveSize): stacks the current archive with new candidates, keeps the mutually non-dominated set in minimization space, and optionally truncates to maxArchiveSize by crowding distance (boundary points always retained). Enables reporting the best Pareto front found over the whole run, not just the final generation's rank-1 front. - crowdingDistance(..., normalizationBounds=None): new optional argument to normalize objective gaps by a population-level (min,max) range instead of the per-front range, so crowding distances are comparable across fronts and generations. Defaults to the previous per-front behavior. testFrontUtils.py: 50/50 (was 46) — archive empty/merge/maximization/truncation cases and population-normalized vs per-front crowding-distance cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the population-normalized crowding distance into the NSGA-II generation
loop as an opt-in <crowdingDistanceNormalization> choice on the multi-objective
optimizer:
- MultiObjectiveGeneticAlgorithm: new input subnode (enum front|population,
default front) + parsing into self._crowdingNormalization.
- NSGAII: both crowding-distance calls (combined (mu+lambda) generation and the
first generation) pass population-level (min,max) bounds when 'population' is
selected, via the new _crowdingNormalizationBounds helper.
Default ('front') is byte-for-byte identical to prior behavior — verified by the
existing NSGA-II_ZDT1_sbx test still matching its gold. The 'population' option
is exercised end-to-end by a new self-contained test NSGA-II_ZDT1_sbx_norm,
whose deterministic Pareto export (differs from the front-normalized run, as
expected) is committed as gold. Optimizer manual regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…unnable tests The NSGA-II multi-objective framework tests were all skipped here: their specs duplicated output files and their decks referenced NSGA-specific OutStream plot types (NSGAParetoFrontPlot, NSGAFrontAnimation, NSGARankHistoryPlot, NSGACrowdingDistancePlot, NSGAFrontRankAnimation) that live on the separate plotting PR (origin/3_Jimmy_adding_Optimization_plots), not on this algorithm branch. Since those plots, their classes, the Conv* input decks and these test registrations all already exist on the plotting PR, this branch: - Deletes (not comments) the NSGA-specific <Plot> definitions and their IOStep outputs from MinwoRepMultiObjective, MinwoRepMultiObjectiveFeasibleFirst and ZDT1 (standard OptPath/PopulationPlot kept) — clean removal so reviewers see no dead/commented code; the plotting PR reintroduces them on merge. - Removes the duplicated image=/Exists declarations so the tests run (Beale emits no plots, so its bogus plot checks are dropped). - Removes the 5 test registrations that cannot run on this branch (4 Conv* whose input decks live on the plotting PR, and ..CustomSampler which needs a missing samples.csv) so the regression machines don't fail/skip on them. The removed blocks are preserved verbatim in pr2547_round3_roadmap_items.md and on the plotting PR. - Regolds the three decks whose Pareto fronts had never been refreshed after the NSGA-II algorithmic fixes (prior golds kept locally as *.orig); Beale's gold was already current. Result: NSGA-II framework tests go from 0 passing / 9 skipped to 6 passing / 0 skipped / 0 failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r fix test_GA_PARCS_LP (single-objective GeneticAlgorithm, onePointCrossover + randomMutator) was failing because its gold predates "Fix GA mutation and crossover operators" (1b30bd7), which changed randomMutator's behavior after the last PARCS regold (6716129). The fixed operators produce a slightly different (valid) loading pattern (loc8/loc9), so the gold was stale. This is unrelated to the multi-objective NSGA-II work: the single-objective GA path (singleObjSurvivorSelect) never touches frontUtils / NSGAII / MOGA, the only files those commits changed. Reproduced deterministically in interface-only mode (test_interface_only=True, no PARCS executable needed) and regolded optOut.csv and opt_export_0.csv to the corrected result; opt_export.csv was already current. The only run-to-run variation is ProbabilityWeight-* metadata column ordering, which the UnorderedCSV differ matches by name. Test now passes (1/1). Prior golds kept locally as *.orig. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the tested updateParetoArchive core into the multi-objective optimizer as an opt-in <paretoArchive> feature so NSGA-II can report the best non-dominated front found over the whole run, not just the final generation's rank-1 front. NSGA-II's (mu+lambda) elitism preserves rank-1 between consecutive generations, but crowding truncation can still drop a previously found rank-1 point when the front exceeds the population size; the archive retains it. - MultiObjectiveGeneticAlgorithm: new <paretoArchive> (BoolType, default False) with an optional maxSize attribute; parsed into self._paretoArchiveEnabled/_paretoArchiveMaxSize. - _collectOptPointMulti: when enabled, _mergeIntoParetoArchive accumulates the current rank-1 front into a persistent archive (carrying decision vars, minimization-space objectives, fitness, constraints and outputs so the exported front keeps all its columns), then _applyParetoArchiveToMultiBest overwrites multiBest* from the archive (ranks all 1, crowding distance recomputed). Merge uses frontUtils.updateParetoArchive with optional crowding-distance truncation to maxSize. - Default off => byte-identical to before (verified: NSGA-II_ZDT1_sbx unchanged). Validation: - testFrontUtils.py 54/54: added the iterative cross-generation accumulation contract the archive relies on (remembers earlier-generation points, collapses to a later dominator). - New self-contained NSGA-II_ZDT1_sbx_archive (maxSize=50): deterministic; its reported front weakly dominates every point of the no-archive run and retains one extra non-dominated solution (7 vs 6) that crowding truncation had dropped. - NSGA-II framework suite 7/7. Optimizer manual regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the two standard distance-based multi-objective quality indicators to complement the existing hypervolume indicator, completing the indicator harness called for in the deep-review benchmark plan: - generationalDistance(obtained, reference): mean nearest-neighbour distance from each obtained point to the reference (true) Pareto front (convergence). - invertedGenerationalDistance(obtained, reference): mean nearest-neighbour distance from each reference point to the obtained front (convergence + coverage; the standard indicator when the true front is known). Both share a vectorized _meanNearestDistance helper. testFrontUtils.py 62/62: identical-front (=0), single-point (=5), asymmetric GD vs IGD cardinality cases, and a demonstration against the analytic ZDT1 front (obj2 = 1 - sqrt(obj1)) showing dense-self IGD = 0 and a coarse on-front approximation within a small coverage bound. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add epsilon-constrained dominance (Takahama & Sato) as an alternative constraint-handling mode for the multi-objective ranking: total constraint violations up to epsilon are treated as feasible, so near-boundary solutions compete on objectives instead of being strictly dominated by fully feasible ones. This can improve exploration along active constraints. - frontUtils: thread an epsilon argument (default 0.0 = strict Deb constrained dominance) through _dominatesForMinimization -> _fastNonDominatedSortConstrained -> _rankNonDominatedFrontiersConstrained -> rankNonDominatedFrontiers. - MultiObjectiveGeneticAlgorithm: opt-in <constraintEpsilon> (FloatType, default 0.0) parsed into self._constraintEpsilon; NSGAII passes it to both non-dominated-sorting calls. - Default 0.0 => byte-identical to before (verified: NSGA-II_MinwoRepMultiObjective unchanged). Validation: - testFrontUtils.py 65/65: epsilon=0 keeps strict feasible-first ranking [1,2,3]; epsilon=2 lets a slightly-infeasible point with better objectives outrank a feasible one -> [2,1,3]. - New self-contained constrained test NSGA-II_MinwoRepMultiObjectiveEps (constraintEpsilon=2.0): deterministic, and its Pareto export differs from the epsilon=0 baseline, confirming the relaxed dominance changes the search end-to-end. - NSGA-II framework suite 8/8. Optimizer manual regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an opt-in adaptive mutation schedule: when <adaptiveMutation> is True the mutation probability is annealed linearly across the generation budget from the configured <mutationProb> (early, exploratory) down to a final value (late, exploitative). This is a simple, well-understood self-adaptation that broadens exploration early and refines the Pareto front late. - MultiObjectiveGeneticAlgorithm: opt-in <adaptiveMutation> (BoolType, default False) with an optional 'final' attribute (defaults to 1/nVariables, the customary per-gene rate); parsed into self._adaptiveMutation / self._adaptiveMutationFinal. - NSGAII: _effectiveMutationProb anneals over (counter-1)/(limit-1) and feeds the mutator; with adaptiveMutation off it returns the constant mutationProb (byte-identical to before, verified by the unchanged ZDT1_sbx / ZDT1_sbx_norm / ZDT1_sbx_archive tests). Validation: new self-contained NSGA-II_ZDT1_sbx_adaptmut (final=0.05) is deterministic and its Pareto export differs from the constant-mutation baseline, confirming the schedule takes effect end-to-end. NSGA-II framework suite 9/9; frontUtils unit 65/65. Manual regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…SGA-II Add an alternative constraint-handling mode that stochastically balances objective progress against feasibility, as a complement to the strict feasible-first rule and the epsilon-constrained relaxation. - MultiObjectiveGeneticAlgorithm: new <stochasticRanking> (BoolType, default False) with an optional pf attribute (default 0.45); parsed into self._stochasticRanking / self._stochasticRankingPf. - NSGAII._rankingConstraintVals: each generation, when enabled, draws a Bernoulli(pf) coin; with probability pf the non-dominated sort is run by objectives only (constraintVals=None) and otherwise by Deb constrained dominance. Applied at both ranking sites (combined (mu+lambda) and first generation). This adapts the per- comparison Runarsson & Yao idea to NSGA-II's set-based ranking in a cycle-safe, deterministic-per-seed way (one coin per generation). - Default off => unchanged (verified: constrained MinwoRepMultiObjective and the epsilon variant still match their golds). Validated end-to-end by new self-contained NSGA-II_MinwoRepMultiObjectiveStoch (deterministic; differs from the strict-ranking baseline, exploring infeasible regions on objective-only generations as expected). Building-block ranking paths (with/without constraints) are already unit-tested in testFrontUtils.py. Optimizer manual regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These two opt-in NSGA-II options previously had only end-to-end framework coverage (NSGA-II_ZDT1_sbx_adaptmut, NSGA-II_MinwoRepMultiObjectiveStoch); add direct unit tests for the per-generation decision logic that lives on the optimizer: - NSGAII._effectiveMutationProb: disabled returns the constant mutationProb; enabled anneals linearly from the initial probability (generation 1) to final (last generation), with the midpoint interpolated and final defaulting to 1/nVariables. - NSGAII._rankingConstraintVals: disabled and pf=0 keep the constraint values (strict ranking); pf=1 returns None so the generation ranks by objectives only. testNSGAIIOptions.py 8/8; registered in the Optimizers unit-test driver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n-tool PR) generationalDistance / invertedGenerationalDistance require a reference (true) Pareto front, which real RAVEN optimization problems almost never have, so they are not usable as runtime indicators and were wired into nothing. (They are also distinct from, and name-collide with, the GA's reference-free Hausdorff/Delta-p convergence metrics _GD/_ahd used by the AHDp/AHD/HDSM convergence checks, which are unchanged.) Their legitimate use is offline validation/comparison of optimization results via a surrogate reference set (the non-dominated union of the runs being compared) using Pareto-compliant indicators (IGD+, hypervolume). That is a separate post-processing feature and is deferred to its own PR/issue rather than shipped unused here. testFrontUtils.py: 56/56 (removed the 9 GD/IGD checks). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wangcj05
left a comment
There was a problem hiding this comment.
@Jimmy-INL I have few comments for you to consider.
@W0lfShAd0w Please review few modules that I tagged you.
| popSize = population.sizes['chromosome'] | ||
| if kwargs['kSelection'] > popSize: | ||
| mh.error('parentSelectors', ValueError, 'Tournament size cannot be greater than population size') | ||
| candidatePositions = list(range(popSize)) | ||
| if not kwargs['isMultiObjective']: | ||
| # Single-objective case | ||
| if not fitnessProvided and nParents > 0: | ||
| mh.error('parentSelectors', ValueError, "Fitness must be provided for single-objective selection") | ||
| else: | ||
| fitness = kwargs['fitness'] | ||
| fitness = fitVals | ||
|
|
||
| allSelected = set() | ||
| for i in range(nParents): | ||
| matrixOperationRaw = np.zeros((kwargs['kSelection'], 2)) | ||
| selectChromoIndexes = list(set(population.indexes['chromosome']) - allSelected) | ||
| selectedChromo = randomUtils.randomChoice(selectChromoIndexes, size=kwargs['kSelection'], | ||
| replace=False, engine=None) | ||
| selectedChromo = np.asarray(randomUtils.randomChoice(candidatePositions.copy(), | ||
| size=kwargs['kSelection'], | ||
| replace=False, |
There was a problem hiding this comment.
could you explain the changes in these lines? It seems to me the candiatePositions between a range, while the previous implementation using an intercept set to compute.
There was a problem hiding this comment.
The candidatePositions = list(range(popSize)) replaces the old set(population.indexes['chromosome']) - allSelected for two reasons. (1) Bug fix: the previous code drew candidates from the chromosome index labels but then read winners by position via population.values[...], which mismatches whenever the index isn't 0…N-1 — that's the "indexing fix" in commit 10678c4. Using range(popSize) keeps selection positional, consistent with .values. (2) Correct semantics: tournament selection should sample without replacement within a tournament but with replacement across parent slots. The old allSelected exclusion made winners ineligible for later tournaments (without-replacement across tournaments), which is nonstandard and errors out as nParents approaches popSize. The new docstring documents the intended behavior. @wangcj05, Please let me know if this makes sense.
| self.population = offSprings | ||
| self.fitness = offSpringFitness | ||
| self.constraintsV = g | ||
| pass |
There was a problem hiding this comment.
Could you explain why you removed previous implementation here?
There was a problem hiding this comment.
Multi-objective survivor selection was relocated into MultiObjectiveGeneticAlgorithm._resolveNewGeneration when single/multi-objective GA paths were separated; multiObjSurvivorSelect had become a no-op stub nothing calls, so I removed it rather than leave dead code.
There was a problem hiding this comment.
Good catch — it wasn't. The population plot was empty because GA_discreteIntWithReplacement.xml indexed the population/parallel plots (and the opt_export DataObject) on trajID, which is constant (0) for a single-trajectory run — so every chromosome collapsed onto one "Batch #" and nothing showed. Every other GA test correctly indexes on batchId (the per-generation counter). I changed trajID → batchId in the three spots and regenerated the gold. The population plot now shows the min/avg/max band for each variable evolving across generations 1–5, with planValue converging upward — the expected behavior. regolded
There was a problem hiding this comment.
very interesting behavior for this plot.
There was a problem hiding this comment.
I know. looks weird.
The sawtooth is expected for a population-based optimizer. The opt_path plots every individual evaluated each generation (10 per batch, 10 generations), so within each generation the ans values span the population's range — producing the ramp — and reset at the next generation. It's not a single-point trajectory. The optimization itself is healthy: the per-generation best objective is monotonically non-increasing (14 → 13 → 10), elitism is preserved, and the population mean converges from ~22.7 to ~11.4, ending at the minimum of 10. So the "interesting" shape is just the per-generation population spread, not an artifact or regression.
…he repeated function
Untrack the local-only ZDT1_user-initials NSGA-II input that was committed unintentionally and gitignore it so it stays on disk but out of the PR. It was never registered in the tests spec, so no test entry to remove. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tial, And making NSGA-II and III a type under MOGA
Pull Request Description
What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)
#2542
What are the significant changes in functionality due to this change request?
This splits Ga into several classes that will allow similar implementations of other GA algorithms.
For Change Control Board: Change Request Review
The following review must be completed by an authorized member of the Change Control Board.
<internalParallel>to True.raven/tests/framework/user_guideandraven/docs/workshop) have been changed, the associated documentation must be reviewed and assured the text matches the example.