Skip to content

Analysis and Visualization

VAMOS includes optional post-processing helpers for statistical comparison, visualization, and multi-criteria decision making (MCDM). These tools are useful after optimization; they are not required by the core optimization runtime.

Experimental surface in VAMOS 1.4.0

vamos.ux.api, statistical analysis, visualization, and MCDM helpers are experimental. They are supported for evaluation, but may change incompatibly in a minor release. See Stability and versioning.

Install the analysis extras

python -m pip install "vamos-optimization[analysis]==1.4.0"

For environment setup and other extras, see Installation.

Statistical comparison

For real saved observations, follow T07 and T08, or inspect their recorded report. That two-problem example uses descriptive summaries and ranks only; it does not justify an inferential test.

The public vamos.ux.api facade exposes non-parametric helpers for comparing algorithms across benchmark problems.

The statistical matrix has shape (n_problems, n_algorithms):

  • each row is one benchmark problem (the paired experimental block);
  • each column is one algorithm;
  • each cell is one pre-defined summary of the stochastic repeats for that problem/algorithm pair, such as the median hypervolume across seeds.

Do not put individual seeds in separate rows merely to increase the sample size. Seeds are stochastic replicates of a problem/algorithm cell, not additional benchmark problems. Decide the per-cell estimator before looking at the statistical result.

The matrix below is synthetic illustration of the API, not results from the recorded study. The example uses hypervolume, so larger values are better. For a metric such as IGD+ where smaller values are better, pass higher_is_better=False.

import numpy as np

from vamos.ux.api import friedman_test, pairwise_wilcoxon

algorithm_names = ["NSGA-II", "MOEA/D", "SPEA2"]
scores = np.array(
    [
        [0.85, 0.82, 0.84],
        [0.79, 0.75, 0.77],
        [0.91, 0.88, 0.89],
        [0.72, 0.70, 0.71],
        [0.81, 0.78, 0.80],
        [0.87, 0.83, 0.85],
    ],
    dtype=float,
)

friedman = friedman_test(scores, higher_is_better=True)
print(f"Friedman statistic: {friedman.statistic:.3f}")
print(f"Friedman p-value: {friedman.p_value:.4f}")
print("Average ranks:", dict(zip(algorithm_names, friedman.avg_ranks, strict=True)))

comparisons = pairwise_wilcoxon(
    scores,
    algorithm_names,
    higher_is_better=True,
)
for comparison in comparisons:
    print(comparison.algo_i, comparison.algo_j, comparison.p_value)

friedman_test(...) returns a FriedmanResult with statistic, p_value, ranks, and avg_ranks. pairwise_wilcoxon(...) returns a list of WilcoxonResult objects with algo_i, algo_j, statistic, and p_value.

p_value is each pair's raw Wilcoxon p-value.

VAMOS 1.2.0: - Holm adjustment: each WilcoxonResult also has p_adjusted, Holm's step-down adjustment over all the pairs of the call, and significant, which is p_adjusted <= alpha. vamos bench writes the same values to statistics.json (p_value_holm and significant). Report the procedure you use; for another correction, start from the raw p_value. - Friedman input: friedman_test(...) raises ValueError for fewer than 3 algorithms, fewer than 2 problems, or a NaN score. - Before: in VAMOS 1.1.0, pairwise_wilcoxon ignored alpha and returned only raw p-values. friedman_test let SciPy's error through for two algorithms and returned NaN for a NaN score.

VAMOS 1.2.0: pairwise_wilcoxon(...) counts two equal scores as a tie (Pratt's treatment of zero differences), including two infinite penalties such as the +inf IGD+ of runs without feasible solutions. An infinite score against a finite one is the largest difference. VAMOS 1.1.0 dropped a pair of equal infinities from the test and warned about an invalid subtraction. Two algorithms equal on every problem, for example infeasible everywhere, get statistic 0 and p-value 1 without calling SciPy, whose wilcoxon raises on that input up to version 1.12 and divides by zero after it.

Critical-distance plot

plot_critical_distance(...) visualizes average ranks and, when n_problems is supplied, a Nemenyi-style critical-distance bar. It does not infer or draw significance cliques for you.

VAMOS 1.2.0: the bar is the Nemenyi critical difference CD = q_alpha * sqrt(k * (k + 1) / (6 * N)) for the k plotted algorithms and N = n_problems. The critical value q_alpha depends on k: it comes from Demšar (2006), "Statistical comparisons of classifiers over multiple data sets", JMLR 7, Table 5, for alpha 0.05 and 0.10 with up to ten algorithms, and otherwise from the studentized range quantile with infinite degrees of freedom divided by √2. alpha may be any value strictly between 0 and 1. VAMOS 1.1.0 and earlier used the four-algorithm value for every k, so their bars, including the cd_<metric>.pdf plots of vamos bench, are too short with more than four algorithms and too long with two or three.

The plotting helpers return a Matplotlib Axes; save the figure through that object instead of passing a filename argument.

from vamos.ux.api import plot_critical_distance

ax = plot_critical_distance(
    friedman.avg_ranks,
    algorithm_names,
    n_problems=scores.shape[0],
    show=False,
)
ax.figure.savefig("cd_plot.png", dpi=200, bbox_inches="tight")

Pareto-front visualization

Pass the objective matrix you want to display. plot_pareto_front_2d(...) expects exactly two objectives and plots the supplied rows; it does not perform an additional non-dominated filtering step.

from vamos.ux.api import plot_pareto_front_2d

F = np.array(
    [
        [0.20, 0.90],
        [0.35, 0.65],
        [0.55, 0.45],
        [0.80, 0.25],
    ],
    dtype=float,
)

ax = plot_pareto_front_2d(
    F,
    labels=("f1", "f2"),
    title="Example objective front",
    show=False,
)
ax.figure.savefig("front.png", dpi=200, bbox_inches="tight")

The same facade also exposes plot_pareto_front_3d, plot_parallel_coordinates, and plot_hv_convergence.

VAMOS 1.2.0: plot_pareto_front_2d and plot_pareto_front_3d draw minimization axes, with values increasing away from the origin, so the ideal corner is at the lower left as in the other VAMOS front plots. Pass invert_axes=True for the reversed view. VAMOS 1.1.0 and earlier always inverted every axis, which put the ideal corner at the upper right and made a convex front look concave.

MCDM: selecting one solution from a front

The public MCDM helpers in vamos.ux.api are:

  • weighted_sum_scores(F, weights) — weighted scalarization;
  • tchebycheff_scores(F, weights, reference=None) — weighted Chebyshev distance to a reference point (the component-wise minimum by default);
  • reference_point_scores(F, reference) — Euclidean distance to a supplied reference point;
  • knee_point_scores(F) — a geometric knee heuristic for 2D fronts only.

They return an MCDMResult containing scores, best_index, and best_point. The scoring convention is minimization: the selected index is the one preferred by the helper under the supplied objectives/preferences.

VAMOS 1.2.0: knee_point_scores keeps only the non-dominated points and joins the two extreme points of that set with a chord. The knee is the point farthest from the chord on the side of the ideal point (Das, 1999; Deb and Gupta, 2011), and each score is minus that signed distance. Dominated points score inf. A concave or linear front has no point on that side, so it has no knee: the helper emits a UserWarning, best_index is the extreme point with the smallest f1, and Studio's Results page shows the warning. VAMOS 1.1.0 and earlier use the unsigned distance, which on a concave front picks the point farthest from the ideal and can pick a dominated point. The distances use the units of F, so normalize objectives with different scales first.

Weighted methods are sensitive to objective scale. If objectives use different units or magnitudes, normalize them deliberately before applying weights and keep the returned index to recover the original objective vector (and the corresponding decision vector, if you have an aligned X matrix).

from vamos.ux.api import weighted_sum_scores

span = np.ptp(F, axis=0)
safe_span = np.where(span == 0.0, 1.0, span)
scaled_F = (F - F.min(axis=0)) / safe_span
weights = np.array([0.5, 0.5], dtype=float)

choice = weighted_sum_scores(scaled_F, weights)
best_index = choice.best_index
best_objectives = F[best_index]
print("Selected row:", best_index, best_objectives)

Normalization is a modeling choice, not a neutral preprocessing detail. For a publication-quality analysis, state the scaling rule, weights/reference point, metric direction, replicate aggregation rule, and statistical correction (if any) in the experimental protocol.

Landscape analysis

For landscape-analysis workflows such as random walks, autocorrelation, and ruggedness, see the landscape-analysis notebook. The Examples page explains how to obtain notebooks.