Skip to content

Run a reproducible study

A durable study freezes a problem-by-algorithm-by-seed matrix before any task runs. It preserves task identity, attempts, state transitions, and verified references to each canonical run.

For a connected learning sequence, start with T06: a fair comparative study, then T07: quality and budget and T08: results to report. This lifecycle guide remains the detailed reference for planning, execution, inspection, resume and retry.

The important step is not merely calling StudySpec(...): it is deciding what comparison you intend to make, fixing that design before execution, checking its cost, and keeping every reported row traceable to the run that produced it.

For a bounded repository-checkout example that you can run unchanged, use the Run a reproducible study executable journey.

Start from an experimental question

Suppose you want to compare NSGA-II and MOEA/D on ZDT1 and ZDT2 under the same evaluation budget and the same explicit seed schedule.

The maintained teaching journey uses this matrix:

Factor Values
problems zdt1, zdt2
algorithms nsgaii, moead
seeds 0, 1
maximum evaluations per task 80
population size 20
kernel backend numpy

That is 2 × 2 × 2 = 8 planned tasks and a maximum planned budget of 8 × 80 = 640 objective evaluations.

The numbers are intentionally tiny so the example remains runnable in documentation CI. Two seeds and 80 evaluations are not evidence that one algorithm is better than another. For a scientific study, choose the replication count, evaluation budget, indicators, tuning protocol, and statistical analysis according to the question and observed variability rather than copying tutorial values.

1. Define the complete matrix explicitly

from vamos import StudySpec

spec = StudySpec(
    problems=["zdt1", "zdt2"],
    algorithms=["nsgaii", "moead"],
    seeds=[0, 1],
    max_evaluations=80,
    pop_size=20,
    engine="numpy",
    eval_strategy="serial",
    on_error="continue",
)

StudySpec requires the seed sequence explicitly. All defaults selected while resolving the study are frozen into each task's resolved run specification before the study is published.

Using the same seed list for every algorithm makes the random-seed schedule explicit and prevents accidentally comparing different sets of runs. It does not by itself create common random numbers or a statistically paired experiment: different algorithms can consume randomness differently.

The example holds population size constant for clarity. That is not a universal rule for algorithm comparison. If algorithms need different scientifically justified settings, freeze those settings or their tuning protocol explicitly rather than forcing an inappropriate common configuration.

2. Plan before spending the budget

plan_study(...) resolves the matrix without creating a study or executing an optimization:

from vamos import plan_study

preview = plan_study(spec, output="studies/comparison")

print(preview.status)
print(preview.task_count)
print(preview.total_evaluation_budget)
print(preview.problem_ids)
print(preview.algorithm_ids)
print(preview.seeds)

For the teaching matrix, task_count is 8 and total_evaluation_budget is 640.

Use this stage to catch accidental scope growth before a campaign begins. If you add one more algorithm to a study with 10 problems and 30 seeds, you have added 300 tasks; the plan makes that multiplication visible before execution.

plan_study(...) is read-only. Its output-path check is advisory and does not reserve the destination, so another process could still occupy that path before creation.

3. Publish the plan, then execute it

from vamos import create_study

study = create_study(spec, output="studies/comparison")
assert study.plan_id == preview.plan_id

completed = study.run()

create_study(...) publishes the durable study and its immutable resolved plan; it does not run an optimization. run() is the separate execution step.

Checking the plan_id is useful when a workflow has a review or approval step between planning and execution: it confirms that the study you created is the same resolved plan you inspected.

VAMOS 1.4.0 executes a durable study sequentially under a single mutating owner. eval_strategy belongs to evaluation inside an individual optimization task; it does not make the study itself a concurrent multi-owner scheduler.

VAMOS 1.2.0: a study executes tasks with serial evaluation only. StudySpec still accepts eval_strategy="multiprocessing" or "dask", but plan_study and create_study now reject them with UNSUPPORTED_EVALUATION_BACKEND, and reject negative seeds with INVALID_STUDY_SPEC; nothing is published. In 1.1.0 those plans were reported ready and every task then failed. Use vamos.optimize(..., eval_strategy=...) for parallel evaluation of a single run.

On Windows without long paths enabled, keep the output path short: the staging directory and the tasks/<task hash>/ folders add about 130 characters to it. VAMOS 1.2.0: when the path length makes creation fail, the actual field of the ATOMIC_PUBLICATION_FAILED error says so; see Troubleshooting.

4. Inspect operational state before interpreting results

report = completed.inspect()

print(report.state)
print(report.counts)
print(report.verified_run_count)
print(report.issues)

inspect() answers operational questions: which tasks succeeded, which failed or remain unfinished, whether referenced run metadata could be verified, and whether recovery work is required.

VAMOS 1.3.0: load_study(path) verifies every referenced run completely, so one damaged or missing run makes it raise. To inspect such a study, load it with verify="metadata": the load checks each run's manifest and artifact hashes without reading its arrays, and inspect() and summarize() report the damaged run as an issue instead of failing. run(), resume() and retry() still reload with complete verification and refuse damaged evidence.

from vamos import load_study

report = load_study("studies/example", verify="metadata").inspect()
print(report.issues)

A trusted study that needs larger bounds, such as long metadata strings or very large result arrays, takes them once: load_study and create_study accept limits= (a StudyLoadLimits for the study documents) and run_limits= (a LoadLimits for the study's runs), and the returned handle keeps both for every reload by run(), resume(), retry(), cancel(), inspect() and summarize(). VAMOS 1.2.1 and earlier had no tolerant load in the public API, accepted no run limits, and reloaded with the default limits, so a study created with larger limits could be loaded but not inspected or run.

A completed study is not automatically a scientifically adequate study. Operational success means the planned tasks and their evidence are in a coherent durable state; it does not establish convergence, sufficient replication, a meaningful indicator, fair tuning, statistical significance, or practical importance.

5. Trace every summary row to its run

StudySummary has one deterministic row for every planned task. The row keeps the experimental factors and the run evidence together:

summary = completed.summarize()

for row in summary.rows:
    print(
        row.problem_id,
        row.algorithm_id,
        row.seed,
        row.evaluation_budget,
        row.state,
        row.evaluations,
        row.selected_run_id,
        row.run_manifest_path,
    )

For a successful task, selected_run_id identifies the selected successful run and run_manifest_path points to its canonical manifest inside the study directory. The summary also carries fields such as run_manifest_sha256, run_metadata_available, termination information, runtime, and failure information when applicable.

This is the key provenance chain:

experimental factor combination
        ↓
planned task_id
        ↓
selected attempt
        ↓
canonical run_id
        ↓
run manifest and numerical artifacts

Do not discard problem_id, algorithm_id, seed, or the run identifiers when exporting results for downstream analysis. A table of indicator values without provenance is much harder to audit or reproduce.

6. Separate durable evidence from statistical conclusions

A study summary is a deterministic projection of the planned tasks and canonical run evidence. It is not an automatic leaderboard and does not decide that one algorithm "wins".

Before making comparative claims, define the analysis separately. Typical questions include:

  • which performance indicator is appropriate for the problem and objective count;
  • whether all algorithms received comparable computational budgets;
  • how algorithm-specific hyperparameters were chosen;
  • how many independent runs are needed to characterize stochastic variability;
  • what statistical comparison and effect-size reporting are appropriate across problems;
  • whether failed or incomplete tasks must be resolved before analysis.

The durable study layer gives that analysis a stable evidence base. It deliberately does not replace experimental-design judgment.

Python lifecycle in one block

from vamos import StudySpec, create_study, load_study, plan_study

spec = StudySpec(
    problems=["zdt1", "zdt2"],
    algorithms=["nsgaii", "moead"],
    seeds=[0, 1],
    max_evaluations=80,
    pop_size=20,
    engine="numpy",
    eval_strategy="serial",
    on_error="continue",
)

preview = plan_study(spec, output="studies/example")  # read-only
study = create_study(spec, output="studies/example")  # no optimization yet
assert study.plan_id == preview.plan_id
completed = study.run()                               # sequential study execution

report = completed.inspect()
summary = completed.summarize()
print(preview.task_count, preview.total_evaluation_budget)
print(report.state, report.counts, report.verified_run_count)
print(len(summary.rows))

loaded = load_study("studies/example")                # data-only

Study is an immutable snapshot. A mutating method returns a newly loaded snapshot; the previous object does not update in place. Inspection and summary verify and project persisted metadata without materializing result arrays or executing components.

CLI lifecycle

The configuration file contains one JSON object whose keys match StudySpec:

{
  "problems": ["zdt1", "zdt2"],
  "algorithms": ["nsgaii", "moead"],
  "seeds": [0, 1],
  "max_evaluations": 80,
  "pop_size": 20,
  "engine": "numpy",
  "eval_strategy": "serial",
  "on_error": "continue"
}
vamos study plan study.json --output studies/example --json
vamos study create study.json --output studies/example --json
vamos study run studies/example --json
vamos study inspect studies/example --json
vamos study summarize studies/example --format csv --output artifacts/studies/example.csv --json

Plan first and review the task count and total budget before create or run. JSON mode writes one vamos.study-command-result document to stdout; diagnostics and warnings go to stderr.

Resume and retry

After a crash or a killed process, reload the directory, resume the pending and interrupted work, and retry the failed tasks if there are any:

from vamos import load_study

study = load_study("studies/example").resume()
if study.inspect().retryable_failed_work:
    study = study.retry(failed_only=True)

VAMOS 1.3.0: a study whose process was killed after it wrote its first journal event and before its root checkpoint recorded that event loads and resumes like one stopped at any later event. VAMOS 1.2.1 and earlier rejected it with CHECKPOINT_JOURNAL_INCONSISTENCY.

resume() does nothing when no task is pending or interrupted. retry() needs a failed task that can be retried. On a study whose tasks all succeeded, it raises RetryNotAllowedError (a StudyError) with reason TERMINAL_TASK_NOT_RETRYABLE, and vamos study retry exits with code 4. Check retryable_failed_work first, as above.

The CLI equivalents are:

vamos study resume studies/example --json
vamos study resume studies/example --retry-failed --json
vamos study retry studies/example --failed --json

vamos study inspect studies/example --json reports retryable_failed_work and retryable_task_ids. vamos study resume --retry-failed, or resume(retry_failed=True) in Python, resumes and retries in one step, and does nothing when no task is left to run.

Ctrl+C is different. It cancels the study: the running attempt is recorded as cancelled, and cancelled is final, so resume() and retry() refuse the study; create a new one for the work left. A process that is killed or crashes leaves the study running, and resume() reconciles it and continues. In Studio, Stop on the Batch Comparison page ends the worker without cancelling the study, which can then be resumed. VAMOS 1.3.0: inspect() and the vamos study commands no longer suggest resuming a cancelled or failed study. VAMOS 1.2.1 and earlier told vamos study run users to run vamos study resume after Ctrl+C, which then failed with INVALID_STATE_TRANSITION. A Ctrl+C that arrives before execution starts, or while vamos study inspect or summarize reads the study, changes nothing: the command reports interrupted and points to vamos study inspect without claiming that the study was cancelled.

Retry is explicit and bounded by max_attempts_per_task. VAMOS does not mutate the immutable study plan or silently substitute current convenience defaults.

A retry creates new attempt evidence for the same planned task; it does not quietly turn the study into a different experimental matrix. When analyzing a retried study, retain the selected-attempt and run provenance recorded by the summary.

VAMOS 1.2.0: a failed task records its cause. For a VAMOS error, its message (in the task, the attempt and the failed run) is the error type and its text on one line, up to 500 characters. Absolute paths are replaced by <path>, including paths with spaces, and the field of a typed artifact error is included. For any other exception, only its type is recorded, because its text may hold credentials, tokens or personal paths. The process that runs the study logs the same message at error level under vamos.experiment.study, with the type, file name, line and function where each exception in the chain was raised; it never logs the exception text itself. A failure that the same task would repeat is not retryable, and retry refuses it with NONRETRYABLE_FAILURE: a ConfigurationError or problem error, an evaluation that writes non-finite or misshaped objectives or constraints, or a failed reconstruction. Other failures after the optimization starts stay retryable. In VAMOS 1.1.0 every failure stored the same fixed text with no cause, and any failure after the optimization started counted as retryable, so retry repeated deterministic failures until max_attempts_per_task.

Ownership boundary

VAMOS 1.4.0 permits one mutating owner per study. Do not run run, resume, or retry concurrently against the same directory. Nothing detects a second owner: two processes can run the same task twice, and one of them usually stops with EVENT_APPEND_COLLISION when it finds the journal entry that the other has just written. Execution is sequential; distributed workers, multiprocess ownership, and cross-process cancellation are unsupported. See Known limitations.

VAMOS 1.2.0: reading a study while another process runs it (load_study, inspect(), summarize(), vamos study inspect, or Studio's Batch Comparison page) no longer fails on the files that process is writing. On Windows, the running process waits for a concurrent read to finish instead of stopping with CHECKPOINT_WRITE_FAILED, and the reader waits for a replace to finish instead of failing. A process killed in the middle of a write leaves a .tmp- file next to the document; loading ignores it and resume deletes it. VAMOS 1.1.0 refused to load such a study until the file was removed by hand.

Next steps