Known limitations in VAMOS 1.4.0¶
These limitations are part of the current release scope. They describe unsupported behavior rather than future compatibility promises.
Algorithms and numerical behavior¶
- VAMOS minimizes every objective. Users must transform maximization objectives themselves.
- Objective and constraint values must be finite: a NaN or an infinity stops
the run with
EvaluationError. Give failed evaluations a large finite penalty or a violated constraint. - VAMOS 1.2.0: constraint values from a problem that does not
declare
n_constraintsraiseEvaluationError; they were ignored. - A valid population size may depend on an algorithm's reference directions or weight lattice. Incompatible sizes fail instead of being adjusted silently.
- VAMOS 1.2.0: NSGA-III accepts a population at least as
large as its reference directions (N >= H), as Deb and Jain, jMetal and
pymoo do, and fails only below that. Without
divisionsit uses the complete Das-Dennis lattice with the most directions that fit inpop_size: 91 directions for 100 individuals and three objectives. VAMOS 1.1.0 and earlier truncated the next larger lattice topop_sizedirections, which dropped the corner of the first objective; theref_dirsarchive pruning andtop_k(method="reference_directions")had the same bias. NSGA-III and RVEA with two objectives default to 100 directions, orpop_size, instead of 7. - The strict evaluation budget must be at least the resolved population size.
VAMOS 1.2.0: with the opposition-based initializer
(
initializer: obl), it must be at least twice the population size: the2 * pop_sizecandidates (random points and their opposites) count against the budget, and the chosen ones keep their objective and constraint values instead of being evaluated again. The initializer keeps feasible candidates first, then the best by non-domination rank and crowding distance. VAMOS 1.1.0 evaluated the candidates outside the budget, so a run evaluated2 * pop_sizemore rows than it reported, and chose them by objectives alone, discarding feasible candidates for infeasible ones. - VAMOS 1.2.0: NSGA-III, SMS-EMOA and SPEA2 mate parents two by
two, so they reject the three-parent crossovers
pcx,undx,simplexandde; MOEA/D acceptsdebut rejects the other three. NSGA-II, IBEA, AGE-MOEA and RVEA group parents by the crossover's arity and run all four. For real encoding, NSGA-III, SMS-EMOA, SPEA2 and MOEA/D also reject crossovers and mutations of other encodings, as NSGA-II does. The initializerslhs,scatter,sobol,haltonandoblrequire real encoding in every algorithm. These errors are raised before the first evaluation. VAMOS 1.1.0 failed only at the first generation with a three-parent crossover, ran binary operators on real variables, and gave binary problems fractional initial bits. - Deterministic same-environment runs do not imply bitwise equality across different kernels, BLAS implementations, operating systems, architectures, or optional third-party engines.
Benchmark validation¶
welded_beamis a six-variable mixed design example with its own formulas, materials and stiffeners. Its constraint metadata is fixed, but its formulas are not those of the four-variable welded-beam benchmark thatre32andcre22use: the weld cost lacks the factorl, the normal stress usest b^2fort^2 b, the polar moment and the shear stress differ, the deflection is divided by the stiffener count, and there is no buckling constraint. Its results are not comparable with published welded-beam results; the formulas are listed under Problems.- VAMOS 1.2.0:
lsmop1–lsmop9follow the original LSMOP definition (Cheng et al., 2017), as PlatEMO does. VAMOS 1.1.0 and earlier follow jMetal, with the nonlinear linkage in LSMOP1–4 and a larger distance term in LSMOP9. The Pareto fronts are the same, but LSMOP1–4 and LSMOP9 results from those versions do not correspond to the original problems. - VAMOS 1.2.0:
cec2009_uf3searches the CEC2009 domain[0, 1]^n. VAMOS 1.1.0 and earlier search[-1, 1]for variables 2 to n, so UF3 results from those versions do not correspond to the CEC2009 instance. - VAMOS 1.2.0:
re24searchesx2in[0.5, 50]andre25uses the 19 wire diameters from 0.092, as Tanabe and Ishibuchi's reference implementation does, andre24.csvis the authors' approximated front. VAMOS 1.1.0 and earlier usex2 >= 4in RE24, which cuts its front at f₁ = 480.5, and all 42 listed diameters in RE25. RE24 and RE25 results from those versions do not correspond to the reference problems.re22keeps its documented reading of the reference's "3,10" as 3.10. - VAMOS 1.2.0:
cec2009_uf9uses the CEC2009 squared distancey_j^2. VAMOS 1.1.0 and earlier use the multimodal distance of UF5, with the same Pareto front but a different landscape, so UF9 results from those versions do not correspond to the CEC2009 instance. - VAMOS 1.2.0:
rwa7andrwa8search the normalized domain[0, 1]^4of their response surfaces, with the TTmax sign ofre37, and their packaged reference samples are regenerated. VAMOS 1.1.0 and earlier evaluate the surfaces far outside their fitted region, on the physical design ranges, so RWA7 and RWA8 values, fronts and indicators from those versions do not correspond to the published problems. - VAMOS 1.2.0: the inaccurate packaged reference fronts are regenerated. IGD, IGD+, epsilon and normalized HV values that VAMOS 1.1.0 and earlier computed with them are not comparable with later ones:
- Analytic samples: ZDT, DTLZ, WFG1, WFG2, LSMOP9, UF6, UF9, and C-DTLZ and DC-DTLZ at two and three objectives;
- Authors' fronts: the RE fronts that Tanabe and Ishibuchi published;
- Dense sampling with polishing: RE91, RWA3, RWA9 and RWA10.
The reference fronts section
lists the origin of each file.
- DC-DTLZ follows its authors' implementation (Li et al., 2019) by default,
as pymoo does. jMetal, PlatEMO and STEP use other constants in all six
problems, so their DC-DTLZ results are not comparable with those of VAMOS's
defaults. vamos.problems.DCDTLZ takes their constants as parameters. See
Problems.
- Independent algorithm-conformance validation for AGE-MOEA, RVEA and SMPSO
remains pending. Passing API and regression tests does not establish
convergence equivalence with a reference implementation.
- VAMOS 1.2.0: AGE-MOEA's hypervolume on ZDT1, DTLZ2 and DTLZ1
is statistically indistinguishable from jMetal 7.9's over 30 seeds, and a
reference test checks it on ZDT1. VAMOS 1.1.0's AGE-MOEA was significantly
worse, mainly because its mating tournament used NSGA-II crowding instead of
the survival score; see
AGE-MOEA survival score.
RVEA and SMPSO remain pending.
- IBEA normalizes the objectives over the population for both indicators,
as adaptive IBEA does.
- Effect on multimodal problems: far-dominated solutions in early
populations widen the normalization range, which lowers the resolution
among the other solutions. This slows convergence on problems such as ZDT4
and DTLZ1; jMetal's IBEA behaves the same.
- Earlier versions:
- VAMOS 1.1.0 used the epsilon indicator unscaled, so objective units
shaped its selection.
- VAMOS 1.0.2 and earlier also inverted the hypervolume indicator and
counted overflowing contributions as zero; see the
IBEA indicator corrections.
- Overflow: with the normalized indicators, contributions exceed the
float64 range only for kappa below about 1/709, and selection then works
in the log domain.
- Infinite objective values: they give infinite contributions, and
solutions with such contributions are not ordered among themselves.
Backends and optional dependencies¶
- NumPy is the reference path. Numba accelerates selected kernels rather than every operation; MooCore accelerates selected indicators rather than whole algorithms.
- The Numba engine compiles SBX and polynomial mutation only where a one-time
check finds that compiled
powmatchesnp.powerbit for bit. A NumPy build whosenp.poweruses vendor SIMD code fails it, and the Numba engine then keeps the NumPy operators: the same results, but slower (see algorithms). - Explicitly selecting an unavailable backend is an error. VAMOS does not install dependencies during a run, verification, or replay.
- VAMOS 1.2.0: hypervolume archive pruning (
pruning: hvormc_hv) with three or more objectives requiresmoocoreand fails without it. VAMOS 1.1.0 and earlier prune by crowding distance instead, with a single warning; see Stopping and archive. - Optional research frameworks and model-based tuning packages have their own
Python and operating-system constraints. VAMOS 1.2.0: the
research frameworks are no longer an extra (the
researchextra is removed). - VAMOS 1.2.0: the release CI claims Python 3.10 to 3.14 on Linux, and Python 3.12 to 3.14 on Windows and macOS; VAMOS 1.1.0 claimed Python 3.10 to 3.12 on Linux and 3.12 on Windows and macOS. VAMOS 1.3.0: VAMOS requires Python 3.11 or newer, and the release CI claims Python 3.11 to 3.14 on Linux. Other interpreter/platform combinations are unclaimed. The tuning extra is tested only on Linux with Python 3.12.
Run artifacts and replay¶
- Public readers support the 1.0.0 run schema. Internal pre-public formats are unsupported and must be regenerated.
- Exact replay is available only for reconstructable registered built-in components with exact material environment compatibility.
- Custom Python, plugins, cross-backend execution, and best-effort environment matches are not exact-replay targets.
- SHA-256 evidence detects modification; it is not an authenticity signature.
- On Windows without long paths enabled, a run or study cannot be written when its staging paths reach 260 characters (248 for a directory), and the Numba cache fails in deeply nested installations. VAMOS 1.2.0: the error gives the path length and the ways out; see Troubleshooting.
- VAMOS 1.2.0: saving and verification classify the installed VAMOS sources with the same rule.
- Wheel inside another Git project: a Git repository counts as a VAMOS
checkout only when its
src/vamosis the imported package, or when VAMOS is installed in editable mode. A wheel installed in a virtual environment inside another Git project is recorded as a wheel, and VAMOS no longer runsgit statusorgit diffon that project. - Runs saved by 1.1.0 in that layout: they recorded the project's revision as their source, so a later commit to the project blocks their exact replay and the resumption of their studies. Rerun them.
- Editable install without Git: an example is a downloaded source tree
installed with
pip install -e .. Its runs now verify asexactwhile the sources are unchanged; 1.1.0 recorded them as wheels and never verified them as exact. - Packaged data: checkouts and editable installs also record a
fingerprint of the packaged data under
vamos/resources. Changing a TSPLIB instance or a weight file after saving a run therefore blocks its exact replay. Runs saved by 1.1.0 have no such fingerprint and verify as before.
Durable studies¶
- Study mutation is single-owner. Do not run, resume, or retry the same study
concurrently from multiple processes: nothing detects a second owner, so a
task can run twice, and one process usually stops with
EVENT_APPEND_COLLISION. - Ctrl+C cancels a running study, and a cancelled study cannot be resumed; a killed or crashed process leaves it resumable. See Resume and retry.
- Windows offers no directory fsync, so study and run writers skip it there: a power loss just after a write can lose the newest entry of a directory, but never leaves a partial document or run under its final name.
- Study execution is sequential in 1.4.0. Distributed workers, multiprocess ownership, and cross-process cancellation are unsupported.
- VAMOS 1.2.0: study tasks run with serial evaluation only;
planning and creation reject
eval_strategy="multiprocessing"or"dask"withUNSUPPORTED_EVALUATION_BACKEND. - Cancellation is cooperative and local to the process that owns execution. Studio's token-addressed worker stop ends the worker and requires reconciliation on resume; it is not durable cross-process cancellation of the study API.
Analysis and tuning¶
- Statistical analysis, visualization, MCDM helpers, tuning, and racing APIs remain experimental and may change in a minor release.
- Users are responsible for selecting indicators, reference points, sample sizes, and statistical tests appropriate for their scientific claim.
- VAMOS 1.2.0: the Nemenyi critical difference of
plot_critical_distanceand of thecd_<metric>.pdfplots ofvamos benchuses the critical value for the number of algorithms. VAMOS 1.1.0 and earlier use the four-algorithm value for every count, which understates the critical difference with more than four algorithms; see Critical-distance plot. - VAMOS 1.2.0:
knee_point_scoresand Studio's "Knee point" ranking measure the signed distance to the extreme-point chord among the non-dominated points, and warn when the front has no knee. VAMOS 1.1.0 and earlier pick the point farthest from the ideal on a concave front and can pick a dominated point; see MCDM. - VAMOS 1.2.0: the LaTeX tables of
vamos benchmark differences with a rank-sum test per problem and say when the number of runs cannot reachalpha. VAMOS 1.1.0 paired runs by seed in a signed-rank test, so with five or fewer seeds its tables showed no significant difference, and it put the worst value in bold for some aliases of the epsilon and Hausdorff indicators; see Other subcommands.
Studio¶
- Studio is experimental and is intended for trusted local use. It is not a multi-user hosted service.
- VAMOS 1.2.0: the Statistical Comparison counts a run without feasible solutions as the worst IGD+ (∞). VAMOS 1.1.0 and earlier leave those runs out of IGD+, which favors algorithms that are rarely feasible; see Statistical comparison.
- Remote binding increases exposure and requires explicit opt-in.
- VAMOS 1.3.0: each Studio launch has an access token, but it travels over plain HTTP, and whoever has the link can use Studio with the account that serves it until the launch ends. To reach Studio from another computer, prefer an SSH tunnel to the loopback port over remote binding; see Installation and launch options.
- VAMOS 1.4.0: Studio runs registered problems only. The Problem Builder, which executed reviewed Python typed into Studio or drafted by an LLM provider, is removed, and VAMOS integrates no LLM provider.
Plugins¶
- Plugin descriptors and custom component interfaces are experimental and are not covered by the 1.x stable API.
- Loading and verification never import recorded plugins.
Static typing¶
- The stable API and strict release scope pass mypy. Full-source typing still has a frozen diagnostic ratchet and is not yet zero-error; new or increased diagnostics fail the release gate.