A small working collection for choosing what to test, understanding what a result means, and tracking progress toward evidence suitable for a particular MS&A decision.
Two protocols; no published benchmark runs. These are starting points for evaluation, not claims about JWM or rankings of physics engines. JWM remains private. A missing result means untested here, not a score of zero.
The pendulum evaluator is implemented for private use. The measured pushing case still needs data preparation and a scorer. The guide below explains the measurements; its motions and plots are teaching examples, not result records.
What are we measuring?
Two different questions, two different references. Explore how a simulation can follow its equations correctly yet still need a separate test against the physical world.
01 / Numerical verification
Does the solver follow the equations?
Ideal pendulum · rigid-body or state-space simulators
Teaching calculation only — not a JWM run or a benchmark score.
Input: angle, length, gravity→Compare: cycle timing and energy→Measure: period error and energy drift
Physical motion is represented by angle and angular velocity. Even this simple model can reveal integration error.The time between matching downward zero crossings gives one period. The benchmark compares that period with its analytical value and checks how far energy strays from its initial value. The red segment illustrates angle disagreement; it is not a scored metric here.
Illustration: nonlinear ideal-pendulum equation, 1 m length, 50° release, gravity 9.81 m/s². Coarse symplectic Euler uses a 0.12 s step; the teaching reference uses RK4 at 0.0025 s. These settings are not the benchmark protocol.
What this establishes: agreement with the specified equations. It does not establish that an ideal pendulum accurately represents a particular real pendulum.
02 / Physical validation
Does the prediction match the experiment?
Planar pushing · contact/friction engines or learned state dynamics
Illustrative paths only — not MIT measurements, JWM results, or benchmark scores.
Input: object, surface, push→Compare: predicted and observed pose→Measure: position and orientation error
Friction and contact assumptions affect both the object’s path and how it turns.Position mismatch and orientation mismatch are separate measurements. Neither a realistic rendering nor one error number tells the whole story.
The paths here are drawn examples, not a contact solver. A real evaluation must use the same push inputs, aligned coordinates and timing, calibrated observations, and an explicit treatment of measurement uncertainty.
What this establishes: agreement with observations under tested conditions, once the protocol is implemented. Good performance on one surface or object does not establish validity elsewhere.
These tests compare physical state trajectories. A video-only world model needs calibrated state extraction and its uncertainty assessed before its output can enter this kind of comparison. Both illustrations start paused, including when reduced motion is preferred.
Can a model predict held-out object motion under a measured pushing action, with uncertainty and preprocessing made explicit?
measured
candidate
0 published runs
The collection deliberately separates numerical correctness from agreement with measurements. Uncertainty calibration, unfamiliar operating conditions, and decision usefulness are not yet covered. Runtime can accompany either test, but only an accuracy-qualified comparison would support a speed–accuracy claim.
Does the implementation approach the prescribed equations as the timestep decreases, without introducing unacceptable energy error?
Why selected. A compact first check with a mathematical reference independent of the simulator. The equations and reference mathematics are established; the amplitudes, run lengths, metrics, and reporting rules below are a proposed jBOK protocol, not an upstream benchmark standard. This checks numerical implementation, not agreement with a real pendulum.
Reference. Undamped point mass on a massless rigid rod: theta'' = -(g/L) sin(theta), released from rest. For amplitude theta0 below pi, T = 4 sqrt(L/g) K(sin(theta0/2)). K uses the modulus convention; libraries that accept the parameter m require m = sin(theta0/2)^2. Independently compute K with the NIST arithmetic-geometric-mean identity. E = 0.5 m L^2 omega^2 + m g L (1 - cos(theta)) is constant.
Reference uncertainty. No experimental uncertainty is represented. The local evaluator cross-checks its binary64 AGM reference against numerical quadrature and a 80-digit power-series calculation. It separately reports a conditional crossing-resolution bound; that bound is not a statistical confidence interval or a bound on solver error.
Scenario. Use m = 1 kg, L = 1 m, prescribed g = 9.81 m/s^2, zero damping and actuation. Run separate release amplitudes 0.1, 1.0, and 2.0 rad with zero initial angular velocity, each for 20 analytical periods. Compare dt = T/100, T/200, T/400, and T/800. Each amplitude/timestep combination is a separate comparison group.
Readiness and remaining work
Data
No external dataset required. Generate the prescribed initial conditions and retain all output trajectories.
Evaluator
Local trajectory evaluator implemented with independent reference checks, complete-grid validation, period/energy metrics, qualified timing, and source hashes. Evaluator source, adapters, and run artifacts remain in the private research workspace; no scores are published.
License
Citations only. No upstream code, text, or dataset is redistributed; any later code reuse requires its own license review.
Protocol, metrics, and limits
Controls
Record solver and implementation revision, floating-point precision, coordinate convention, timestep, substeps, constraint settings, and output sampling. Do not silently substitute the small-angle equation.
Use the same initial state and stop time across implementations. Detect downward zero crossings with the same documented interpolation; discard the first complete cycle and average subsequent full periods. No time warping or post-hoc phase alignment.
Evaluate energy from the recorded angle and angular velocity using the stated formula at every integration step; normalize against initial energy, which is nonzero for all selected amplitudes.
For timing, exclude initialization and compilation; disclose warm-up, repetition count, hardware, threads, synchronization, and timed region. Compare elapsed time only within matching scenarios and hardware conditions.
Show error versus timestep. A finer timestep is not automatically a better result if roundoff dominates; investigate the convergence range instead of assigning a single aggregate trust score.
Metrics; no values have been published
Metric
Unit
Definition
Absolute relative period error
%
100 times abs(mean simulated period - analytical period) / analytical period, using the fixed crossing rule. Report amplitude, timestep, and number of complete cycles.
Maximum absolute energy drift
%
100 times max over the run of abs(E(t) - E(0)) / E(0). This measures deviation from the prescribed conservative system, not real-world accuracy.
Median simulation time
ms
Median wall-clock duration of the documented simulation region after warm-up. Report repeat count and spread alongside this scalar; interpret together with accuracy.
Failure criteria and interpretation
Reject a result as non-comparable if equations, amplitude, timestep, run length, period extraction, or energy convention differ from the declared comparison group.
Mark runs invalid for nonfinite states, missing required crossings, or incomplete output. Investigate a failure to converge under timestep refinement; it is not sufficient to report one favorable timestep.
No universal numerical pass threshold is asserted. Before using a result for a decision, declare that use case's allowable period and energy errors and evaluate all prescribed cases against them.
Limitations
An ideal one-degree-of-freedom oscillator does not test contact, friction, uncertainty calibration, perception, learning, or physical validation.
Period and energy can hide trajectory defects; retain full trajectories for review. A correct period alone does not establish correct dynamics.
No benchmark scores are public. A local trajectory evaluator and a benchmark-specific JWM adapter are implemented for private use; their existence does not establish physical validity or validate a general-purpose solver. Public independent reproduction is not yet available.
Linear harmonic oscillator. Simpler exact trajectory, but omits amplitude-dependent nonlinear period. Switch when: Use first when debugging integration or testing a model that explicitly assumes small angles; keep its results in a separate benchmark.
Measured physical pendulum. Would assess model-to-reality agreement, but introduces damping, geometry, initial-state, and measurement uncertainty. Switch when: Use when physical validation is the question and calibrated apparatus data are available.
Measured planar pushing: contact and friction prediction
Can a model predict held-out object motion under a measured pushing action, with uncertainty and preprocessing made explicit?
Why selected. MIT MCube provides a primary experimental reference with measured motion and contact forces. It complements the pendulum by confronting a simulator with physical measurements. jBOK proposes a narrow, reproducible subset rather than claiming to reproduce the authors' published evaluation. The subset, split, uncertainty treatment, and evaluator must be frozen before scoring.
Reference. MIT MCube planar pushing data contain timestamped pusher/object poses and interaction forces. The project's repeated-push validation set is a candidate for evaluating both prediction error and repeatability. Use measured pusher motion as input and held-out object motion as the target.
Reference uncertainty. Repeated trials expose variability but do not alone establish calibrated measurement uncertainty. Extract sensor, synchronization, pose-processing, and parameter uncertainty from the primary documentation before defining acceptance thresholds; unknown components must remain explicit.
Scenario. Candidate scope: the rect1 object in the MIT catalog's Validation dataset for modeling [2], associated with Bauza and Rodriguez (ICRA 2017): 1 cm straight pushes at 20 mm/s with repeated trials. This is distinct from the catalog's 2016 repeated pushing collection. Confirm the surface metadata and select one surface for the first release. Freeze exact filenames, exclusions, checksums, coordinate transforms, time grid, and calibration/evaluation assignments in a versioned manifest before accepting results.
Readiness and remaining work
Data
Primary catalog and download links verified; archive retrieval, immutable subset, exclusion list, and checksums pending. A published link is not a verified local dataset.
Evaluator
Candidate protocol only. Data adapter, fixed split, uncertainty analysis, and independently checked scorer remain to be built.
License
Dataset redistribution and upstream code reuse terms have not been established for this release. Link to the owners; do not bundle their data or code until resolved.
Protocol, metrics, and limits
Controls
Keep complete repeated trials intact when splitting data. Separate parameter calibration and model selection from final evaluation; never fit friction or uncertainty parameters on evaluation targets.
Distinguish held-out repetitions of familiar conditions from held-out contact locations or directions. Report them as separate comparison groups; success on repetitions does not demonstrate transfer to new conditions.
Replay measured pusher motion. Supply only the initial measured object state, documented geometry, and permitted calibration parameters; do not feed evaluation object trajectories back into the predictor.
Pin preprocessing and resampling rules, orientation unwrapping, coordinate transforms, known bad-record exclusions, and the shared evaluation interval before scoring. Retain raw-to-processed traceability.
Keep geometry, material parameter provenance, action inputs, evaluation trials, timestep, and output grid identical for a comparison. Report per-trial metrics and aggregate with uncertainty across whole trials.
Record implementation revision, precision, hardware, timing region, warm-up, and repetition count. Runtime comparisons require matching execution conditions.
Metrics; no values have been published
Metric
Unit
Definition
Planar position RMSE
mm
Per trial, sqrt(mean of squared Euclidean x/y position error on the frozen time grid), converted to millimeters. Publish the median across held-out trials with trial count and an interval; retain per-trial values.
Orientation RMSE
rad
Per trial, root mean square of the wrapped angular difference in [-pi, pi]. Publish the median across held-out trials with trial count and an interval.
Final position error
mm
Euclidean x/y error at the common final evaluation time, converted to millimeters. Publish the median across held-out trials; do not substitute this for trajectory error.
Median prediction time
ms
Median wall-clock time for predicting one complete held-out trial after warm-up. State trial duration distribution, repeat count, spread, and included operations.
Failure criteria and interpretation
Do not accept scores until the data manifest, license basis, split, preprocessing, and evaluator are pinned. Do not rank records from different manifests or evaluation tracks together.
Reject results that tune parameters on final evaluation targets, omit failed trials without a declared exclusion rule, use target trajectories as prediction inputs, or silently change alignment to reduce error.
Report nonfinite or incomplete predictions and the number of failed trials separately from errors on successful trials.
Declare decision-specific error tolerances before inspecting evaluation outcomes. Physical agreement is limited to the tested conditions and reference uncertainty; no universal pass threshold is claimed.
Limitations
This is a candidate protocol, not a completed benchmark or a claim of JWM validation. Upstream download links are published, but the archive has not been retrieved and pinned for this release.
Planar rigid-object pushing does not establish fidelity for impacts, deformables, fluids, 3D manipulation, or all friction regimes.
Unknown measurement uncertainty and insufficient metadata may limit interpretation even when trajectory errors are small.
Uncertainty-calibration scores are intentionally deferred until interval definitions and the trial split are fixed; low RMSE is not evidence of calibrated confidence.
Sources and adaptations
MIT MCube — Push Dataset — Primary data catalog, formats, geometry, validation collection, download links, and known bad-record list.
GAUGE — project repository — Alternative under consideration; inspected release status, not an adopted dependency or source of benchmark scores.
InternRobotics — GAUGE repository — Alternative under review; inspected README still describes the evaluation toolkit as forthcoming.
Alternatives and when to switch
Analytical Coulomb-friction block slide. Easy to implement and valuable for checking equations, but agreement with a prescribed friction law is not physical validation. Switch when: Use as an additional numerical verification case while measured data are being prepared.
GAUGE. Targets measured physical fidelity across more regimes. Its inspected repository still labels the evaluation toolkit as forthcoming; this release does not rely on it. Switch when: Reassess when a specific task's data, metadata, evaluator, version, and reuse terms can be retrieved and independently checked.
How this collection evolves
Selection favors a clear question, inspectable references, accessible inputs, and a bounded protocol that can be implemented independently. Source attribution identifies the origin of equations or data; the proposed jBOK protocols and selection judgments are our own adaptations, not endorsements by the cited authors.
Before admitting results, pin the protocol and reference versions; record the implementation revision, physical inputs, execution settings, uncertainty treatment, and exclusions; and retain the underlying evidence. Freeze evaluation cases before tuning. Define any acceptance threshold from the intended decision before interpreting results.
We distinguish three kinds of gap:
Implementation gap: an available test has not yet been run on the implementation.
Benchmark gap: the required data, evaluator, or reproducible protocol is unavailable or insufficient.
Research question: available evidence does not settle the scientific question. An empty dashboard cell alone does not establish novelty.
For the methodological rationale, see the compendium and proposed evaluation dimensions. These benchmarks do not provide an aggregate trust score or certify a model for use.