Skip to content

Known limitations in VAMOS 1.4.0

These limitations are part of the current release scope. They describe unsupported behavior rather than future compatibility promises.

Algorithms and numerical behavior

  • VAMOS minimizes every objective. Users must transform maximization objectives themselves.
  • Objective and constraint values must be finite: a NaN or an infinity stops the run with EvaluationError. Give failed evaluations a large finite penalty or a violated constraint.
  • VAMOS 1.2.0: constraint values from a problem that does not declare n_constraints raise EvaluationError; they were ignored.
  • A valid population size may depend on an algorithm's reference directions or weight lattice. Incompatible sizes fail instead of being adjusted silently.
  • VAMOS 1.2.0: NSGA-III accepts a population at least as large as its reference directions (N >= H), as Deb and Jain, jMetal and pymoo do, and fails only below that. Without divisions it uses the complete Das-Dennis lattice with the most directions that fit in pop_size: 91 directions for 100 individuals and three objectives. VAMOS 1.1.0 and earlier truncated the next larger lattice to pop_size directions, which dropped the corner of the first objective; the ref_dirs archive pruning and top_k(method="reference_directions") had the same bias. NSGA-III and RVEA with two objectives default to 100 directions, or pop_size, instead of 7.
  • The strict evaluation budget must be at least the resolved population size. VAMOS 1.2.0: with the opposition-based initializer (initializer: obl), it must be at least twice the population size: the 2 * pop_size candidates (random points and their opposites) count against the budget, and the chosen ones keep their objective and constraint values instead of being evaluated again. The initializer keeps feasible candidates first, then the best by non-domination rank and crowding distance. VAMOS 1.1.0 evaluated the candidates outside the budget, so a run evaluated 2 * pop_size more rows than it reported, and chose them by objectives alone, discarding feasible candidates for infeasible ones.
  • VAMOS 1.2.0: NSGA-III, SMS-EMOA and SPEA2 mate parents two by two, so they reject the three-parent crossovers pcx, undx, simplex and de; MOEA/D accepts de but rejects the other three. NSGA-II, IBEA, AGE-MOEA and RVEA group parents by the crossover's arity and run all four. For real encoding, NSGA-III, SMS-EMOA, SPEA2 and MOEA/D also reject crossovers and mutations of other encodings, as NSGA-II does. The initializers lhs, scatter, sobol, halton and obl require real encoding in every algorithm. These errors are raised before the first evaluation. VAMOS 1.1.0 failed only at the first generation with a three-parent crossover, ran binary operators on real variables, and gave binary problems fractional initial bits.
  • Deterministic same-environment runs do not imply bitwise equality across different kernels, BLAS implementations, operating systems, architectures, or optional third-party engines.

Benchmark validation

  • welded_beam is a six-variable mixed design example with its own formulas, materials and stiffeners. Its constraint metadata is fixed, but its formulas are not those of the four-variable welded-beam benchmark that re32 and cre22 use: the weld cost lacks the factor l, the normal stress uses t b^2 for t^2 b, the polar moment and the shear stress differ, the deflection is divided by the stiffener count, and there is no buckling constraint. Its results are not comparable with published welded-beam results; the formulas are listed under Problems.
  • VAMOS 1.2.0: lsmop1–lsmop9 follow the original LSMOP definition (Cheng et al., 2017), as PlatEMO does. VAMOS 1.1.0 and earlier follow jMetal, with the nonlinear linkage in LSMOP1–4 and a larger distance term in LSMOP9. The Pareto fronts are the same, but LSMOP1–4 and LSMOP9 results from those versions do not correspond to the original problems.
  • VAMOS 1.2.0: cec2009_uf3 searches the CEC2009 domain [0, 1]^n. VAMOS 1.1.0 and earlier search [-1, 1] for variables 2 to n, so UF3 results from those versions do not correspond to the CEC2009 instance.
  • VAMOS 1.2.0: re24 searches x2 in [0.5, 50] and re25 uses the 19 wire diameters from 0.092, as Tanabe and Ishibuchi's reference implementation does, and re24.csv is the authors' approximated front. VAMOS 1.1.0 and earlier use x2 >= 4 in RE24, which cuts its front at f₁ = 480.5, and all 42 listed diameters in RE25. RE24 and RE25 results from those versions do not correspond to the reference problems. re22 keeps its documented reading of the reference's "3,10" as 3.10.
  • VAMOS 1.2.0: cec2009_uf9 uses the CEC2009 squared distance y_j^2. VAMOS 1.1.0 and earlier use the multimodal distance of UF5, with the same Pareto front but a different landscape, so UF9 results from those versions do not correspond to the CEC2009 instance.
  • VAMOS 1.2.0: rwa7 and rwa8 search the normalized domain [0, 1]^4 of their response surfaces, with the TTmax sign of re37, and their packaged reference samples are regenerated. VAMOS 1.1.0 and earlier evaluate the surfaces far outside their fitted region, on the physical design ranges, so RWA7 and RWA8 values, fronts and indicators from those versions do not correspond to the published problems.
  • VAMOS 1.2.0: the inaccurate packaged reference fronts are regenerated. IGD, IGD+, epsilon and normalized HV values that VAMOS 1.1.0 and earlier computed with them are not comparable with later ones:
  • Analytic samples: ZDT, DTLZ, WFG1, WFG2, LSMOP9, UF6, UF9, and C-DTLZ and DC-DTLZ at two and three objectives;
  • Authors' fronts: the RE fronts that Tanabe and Ishibuchi published;
  • Dense sampling with polishing: RE91, RWA3, RWA9 and RWA10.

The reference fronts section lists the origin of each file. - DC-DTLZ follows its authors' implementation (Li et al., 2019) by default, as pymoo does. jMetal, PlatEMO and STEP use other constants in all six problems, so their DC-DTLZ results are not comparable with those of VAMOS's defaults. vamos.problems.DCDTLZ takes their constants as parameters. See Problems. - Independent algorithm-conformance validation for AGE-MOEA, RVEA and SMPSO remains pending. Passing API and regression tests does not establish convergence equivalence with a reference implementation. - VAMOS 1.2.0: AGE-MOEA's hypervolume on ZDT1, DTLZ2 and DTLZ1 is statistically indistinguishable from jMetal 7.9's over 30 seeds, and a reference test checks it on ZDT1. VAMOS 1.1.0's AGE-MOEA was significantly worse, mainly because its mating tournament used NSGA-II crowding instead of the survival score; see AGE-MOEA survival score. RVEA and SMPSO remain pending. - IBEA normalizes the objectives over the population for both indicators, as adaptive IBEA does. - Effect on multimodal problems: far-dominated solutions in early populations widen the normalization range, which lowers the resolution among the other solutions. This slows convergence on problems such as ZDT4 and DTLZ1; jMetal's IBEA behaves the same. - Earlier versions: - VAMOS 1.1.0 used the epsilon indicator unscaled, so objective units shaped its selection. - VAMOS 1.0.2 and earlier also inverted the hypervolume indicator and counted overflowing contributions as zero; see the IBEA indicator corrections. - Overflow: with the normalized indicators, contributions exceed the float64 range only for kappa below about 1/709, and selection then works in the log domain. - Infinite objective values: they give infinite contributions, and solutions with such contributions are not ordered among themselves.

Backends and optional dependencies

  • NumPy is the reference path. Numba accelerates selected kernels rather than every operation; MooCore accelerates selected indicators rather than whole algorithms.
  • The Numba engine compiles SBX and polynomial mutation only where a one-time check finds that compiled pow matches np.power bit for bit. A NumPy build whose np.power uses vendor SIMD code fails it, and the Numba engine then keeps the NumPy operators: the same results, but slower (see algorithms).
  • Explicitly selecting an unavailable backend is an error. VAMOS does not install dependencies during a run, verification, or replay.
  • VAMOS 1.2.0: hypervolume archive pruning (pruning: hv or mc_hv) with three or more objectives requires moocore and fails without it. VAMOS 1.1.0 and earlier prune by crowding distance instead, with a single warning; see Stopping and archive.
  • Optional research frameworks and model-based tuning packages have their own Python and operating-system constraints. VAMOS 1.2.0: the research frameworks are no longer an extra (the research extra is removed).
  • VAMOS 1.2.0: the release CI claims Python 3.10 to 3.14 on Linux, and Python 3.12 to 3.14 on Windows and macOS; VAMOS 1.1.0 claimed Python 3.10 to 3.12 on Linux and 3.12 on Windows and macOS. VAMOS 1.3.0: VAMOS requires Python 3.11 or newer, and the release CI claims Python 3.11 to 3.14 on Linux. Other interpreter/platform combinations are unclaimed. The tuning extra is tested only on Linux with Python 3.12.

Run artifacts and replay

  • Public readers support the 1.0.0 run schema. Internal pre-public formats are unsupported and must be regenerated.
  • Exact replay is available only for reconstructable registered built-in components with exact material environment compatibility.
  • Custom Python, plugins, cross-backend execution, and best-effort environment matches are not exact-replay targets.
  • SHA-256 evidence detects modification; it is not an authenticity signature.
  • On Windows without long paths enabled, a run or study cannot be written when its staging paths reach 260 characters (248 for a directory), and the Numba cache fails in deeply nested installations. VAMOS 1.2.0: the error gives the path length and the ways out; see Troubleshooting.
  • VAMOS 1.2.0: saving and verification classify the installed VAMOS sources with the same rule.
  • Wheel inside another Git project: a Git repository counts as a VAMOS checkout only when its src/vamos is the imported package, or when VAMOS is installed in editable mode. A wheel installed in a virtual environment inside another Git project is recorded as a wheel, and VAMOS no longer runs git status or git diff on that project.
  • Runs saved by 1.1.0 in that layout: they recorded the project's revision as their source, so a later commit to the project blocks their exact replay and the resumption of their studies. Rerun them.
  • Editable install without Git: an example is a downloaded source tree installed with pip install -e .. Its runs now verify as exact while the sources are unchanged; 1.1.0 recorded them as wheels and never verified them as exact.
  • Packaged data: checkouts and editable installs also record a fingerprint of the packaged data under vamos/resources. Changing a TSPLIB instance or a weight file after saving a run therefore blocks its exact replay. Runs saved by 1.1.0 have no such fingerprint and verify as before.

Durable studies

  • Study mutation is single-owner. Do not run, resume, or retry the same study concurrently from multiple processes: nothing detects a second owner, so a task can run twice, and one process usually stops with EVENT_APPEND_COLLISION.
  • Ctrl+C cancels a running study, and a cancelled study cannot be resumed; a killed or crashed process leaves it resumable. See Resume and retry.
  • Windows offers no directory fsync, so study and run writers skip it there: a power loss just after a write can lose the newest entry of a directory, but never leaves a partial document or run under its final name.
  • Study execution is sequential in 1.4.0. Distributed workers, multiprocess ownership, and cross-process cancellation are unsupported.
  • VAMOS 1.2.0: study tasks run with serial evaluation only; planning and creation reject eval_strategy="multiprocessing" or "dask" with UNSUPPORTED_EVALUATION_BACKEND.
  • Cancellation is cooperative and local to the process that owns execution. Studio's token-addressed worker stop ends the worker and requires reconciliation on resume; it is not durable cross-process cancellation of the study API.

Analysis and tuning

  • Statistical analysis, visualization, MCDM helpers, tuning, and racing APIs remain experimental and may change in a minor release.
  • Users are responsible for selecting indicators, reference points, sample sizes, and statistical tests appropriate for their scientific claim.
  • VAMOS 1.2.0: the Nemenyi critical difference of plot_critical_distance and of the cd_<metric>.pdf plots of vamos bench uses the critical value for the number of algorithms. VAMOS 1.1.0 and earlier use the four-algorithm value for every count, which understates the critical difference with more than four algorithms; see Critical-distance plot.
  • VAMOS 1.2.0: knee_point_scores and Studio's "Knee point" ranking measure the signed distance to the extreme-point chord among the non-dominated points, and warn when the front has no knee. VAMOS 1.1.0 and earlier pick the point farthest from the ideal on a concave front and can pick a dominated point; see MCDM.
  • VAMOS 1.2.0: the LaTeX tables of vamos bench mark differences with a rank-sum test per problem and say when the number of runs cannot reach alpha. VAMOS 1.1.0 paired runs by seed in a signed-rank test, so with five or fewer seeds its tables showed no significant difference, and it put the worst value in bold for some aliases of the epsilon and Hausdorff indicators; see Other subcommands.

Studio

  • Studio is experimental and is intended for trusted local use. It is not a multi-user hosted service.
  • VAMOS 1.2.0: the Statistical Comparison counts a run without feasible solutions as the worst IGD+ (∞). VAMOS 1.1.0 and earlier leave those runs out of IGD+, which favors algorithms that are rarely feasible; see Statistical comparison.
  • Remote binding increases exposure and requires explicit opt-in.
  • VAMOS 1.3.0: each Studio launch has an access token, but it travels over plain HTTP, and whoever has the link can use Studio with the account that serves it until the launch ends. To reach Studio from another computer, prefer an SSH tunnel to the loopback port over remote binding; see Installation and launch options.
  • VAMOS 1.4.0: Studio runs registered problems only. The Problem Builder, which executed reviewed Python typed into Studio or drafted by an LLM provider, is removed, and VAMOS integrates no LLM provider.

Plugins

  • Plugin descriptors and custom component interfaces are experimental and are not covered by the 1.x stable API.
  • Loading and verification never import recorded plugins.

Static typing

  • The stable API and strict release scope pass mypy. Full-source typing still has a frozen diagnostic ratchet and is not yet zero-error; new or increased diagnostics fail the release gate.