Research questions and proposed evaluations
The research agenda asks whether explicit assumptions, provenance, uncertainty, and evaluation boundaries can make simulation-based decisions easier to scrutinize as models change.
Proposal status. The evaluations below have not been executed as a public jBOK benchmark suite. This page specifies questions and intended comparisons, not results, validated thresholds, or a universal trust score.
Questions guiding the work
Can explicit provenance help detect inappropriate model reuse?
Motivation. A model can execute successfully while being used outside the conditions that support its predictions. A complete-looking record can also conceal an unexamined assumption.
Proposed comparison. Present the same model-use cases with ordinary documentation and with structured provenance and validity records. Include appropriate uses, deliberately mismatched uses, and incomplete records. Compare detection of consequential mismatches, false alarms, and time required to reach a justified decision.
Current status and next decision. Question formulation. Define a small case set and independent reference judgments before choosing metrics or conducting a study. Demonstrating a populated schema would not answer this question.
Can assurance information survive component substitution?
Motivation. Physical solvers, learned predictors, and hybrid components do not expose identical assumptions or uncertainty. A common record format does not establish equivalent behavior.
Proposed comparison. Substitute independently developed components for a stated task. Track which evidence remains applicable, which must be regenerated, and which claims become unsupported. Compare with ordinary interface documentation and manual review.
Current status and next decision. Proposed protocol. Select one bounded interface and define both its behavioral obligations and the evidence needed to evaluate a replacement. Generalization across model families remains open.
Which evaluations establish which kinds of confidence?
Motivation. Repeatability, physical accuracy, uncertainty calibration, and fitness for a decision answer different questions. Good performance on one should not silently stand in for another.
Proposed comparison. Use cases that deliberately separate these properties: a repeatable but biased model, a calibrated predictor with limited accuracy, and an accurate model used outside its validated regime. Examine whether the reporting framework preserves those distinctions.
Current status and next decision. Conceptual design. Specify the reference outcomes, tolerances, and decision contexts before selecting pass/fail rules. A reference measurement itself has uncertainty.
Proposed evaluation dimensions
| Dimension | Proposed test | What a successful test could establish | What it would not establish |
|---|---|---|---|
| Provenance and integrity | Alter an artifact or omit a required declaration; include valid controls. | Detection of the specified integrity or disclosure failure. | Truth of declarations or physical accuracy. |
| Reproducibility | Repeat a pinned experiment under declared execution conditions. | Exact, tolerance-based, or statistical repeatability within that scope. | Validity outside those conditions. |
| Numerical verification | Compare computed solutions with analytic, manufactured, or independently established numerical references. | Expected numerical behavior and error within the tested cases. | Whether the governing equations adequately represent the physical system. |
| Physical validation | Compare predictions with held-out measurements or justified physical referents, accounting for reference uncertainty. | Agreement for the stated use and evaluated physical regime. | Universal correctness or validity outside that regime. |
| Uncertainty calibration | Compare predicted intervals with held-out outcomes, stratified by relevant conditions. | Empirical coverage and informativeness for the evaluated distribution. | Reliable behavior after arbitrary distribution shift. |
| Validity-boundary behavior | Exercise cases inside and outside declared operating limits. | Detection and handling of the tested boundary crossings. | An assured intervention path unless that path is also tested. |
| Component substitution | Replace a component with an independently developed implementation. | Compatibility under the tested interface and scenarios. | Ecosystem-wide interchangeability. |
| Decision usefulness | Compare judgments or fault detection with and without the framework. | Practical benefit under the evaluated task and study design. | General benefit without further replication. |
These dimensions should remain separately visible. A single aggregate “trust” score could hide a severe weakness behind strong performance on an unrelated dimension.
Before a protocol becomes a benchmark
Each protocol needs a stated claim and scope; references with their uncertainty; realistic baselines; positive and negative controls; metrics and acceptance criteria chosen before interpreting outcomes; separation of development and evaluation cases; and a record of versions, execution conditions, exclusions, and limitations.
For stochastic studies, the design must address repeated runs, dependence between samples, and uncertainty in reported comparisons. For human evaluation, it must specify participant selection and how reference judgments are established. Results would be reported with unsuccessful outcomes and limits, rather than treating a conformance check as proof of decision fitness.
The compendium records the methodological foundations and alternatives. The benchmark register now provides two concrete starting protocols: ideal-pendulum numerical verification and a measured planar-pushing validation candidate. Readiness and remaining work are explicit. The comparison workbench can inspect local result files without publishing them.
Development sequence
- Public synthesis: clarify scope, sources, alternatives, and open questions. Published and evolving.
- Protocol development: select bounded cases, specify comparisons, and revise the protocols in response to critique. The register and workbench are the current step; no completed evaluation is claimed.
- Shareable MVP: release a small, useful artifact and reproduction instructions when the private implementation is ready.
- Comparative evaluation: publish scoped results, limitations, and evidence that others can examine.
- Research publication: develop a submission around a demonstrated contribution and the policies of the chosen venue.
These are dependencies, not promised completion dates. A negative result may change the sequence or narrow the intended scope.
Research log
November 3, 2026 — Pendulum evaluator and visual benchmark guide
Type: evaluation tooling and explanation. Implemented a local pendulum trajectory evaluator with independent reference checks, complete-grid checks, and separate period, energy, and timing metrics. Added a benchmark-specific adapter for the private JWM scheduler and local result import. The adapter supplies its own oscillator model; it does not evaluate an existing general-purpose JWM physics solver.
Added animated teaching diagrams showing model inputs, reference comparisons, and the meaning of the measured errors. These illustrations use teaching calculations or schematic paths, with explicit labels and motion controls.
Public evidence boundary: evaluator code and run artifacts remain private. No experimental scores or physical-validation claim are added to the public register. The measured pushing protocol remains a candidate.
November 3, 2026 — Benchmark register and comparison workbench
Type: protocol design and software tooling. Selected an analytical pendulum case and a measured planar-pushing candidate, with sources, controls, metrics, limitations, and remaining data/evaluator work. Added a comparison interface that accepts local result records without uploading them. The initial public result collection is empty.
What this changes: the ability to inspect evaluation choices and compare compatible future records. What it does not establish: measured JWM performance, validated physical fidelity, uncertainty calibration, or a field-wide research gap.
November 3, 2026 — Public synthesis revision
Type: literature synthesis and editorial clarification. Reframed the site as a developing research compendium; added six sourced recommendations with alternatives and switching conditions; separated intellectual origin from evidence status; outlined proposed evaluations; and retained the previously published JWM history with clearer limits.
What this changes: the public reasoning and its traceability. What it does not add: new experimental data, a public implementation release, completed benchmarks, or independent validation.
August 11, 2026 — Historical implementation-profile snapshot
The previously published JWM v0.81 profile reports an environment-interface integration and replay checks, alongside an unsuccessful PPO learning experiment. These remain dated, author-reported observations with private supporting artifacts; they are not results of the proposed evaluations above.