jBOKResearch compendium
Work in developmentPublic revision

A developing compendium

These entries explain choices for simulation systems used by people and AI agents. They connect established work in physics, computational mathematics, verification, validation, uncertainty quantification, and provenance to proposed engineering practices. The aim is to make the reasoning inspectable and revisable.

Origin describes where a recommendation comes from. Support describes what supports it. Every entry below is a jBOK synthesis of established methods, with literature support for its foundations. None has been empirically validated as a complete jBOK recommendation. The evaluation paragraphs describe future work, not completed experiments. Sources were reviewed for this revision on November 3, 2026.

The selection is deliberately small. It is neither an exhaustive review nor a universal rulebook. Citing a standard does not establish jBOK compliance with it. Applying these ideas to learned, physical, and hybrid world models is a research objective whose limits remain to be tested.

In this edition:

Credibility belongs to a stated use

Origin: jBOK synthesis. Support: literature-supported foundations; proposed application.

Context. A model may be adequate for comparing broad trends yet inadequate for choosing an operating limit. A useful review must identify the decision being made and what would count as an unacceptable error.

Recommended default. Before interpreting a result, record the intended decision, quantities of interest, operating regime, acceptance criteria, and responsible reviewer. Keep verification, validation, uncertainty, and permission to use the result as distinct judgments. Reconsider those judgments when the decision or model changes.

Sources and adaptation. NASA-STD-7009B (2024), §§4.1.1.1–4.1.1.5 and 4.3.1 grounds modeling practice in intended use, criticality, acceptance criteria, and assessment of proposed use. jBOK adopts that framing and proposes carrying a compact use record with experimental artifacts. It does not import NASA's institutional approval structure.

Rationale. This choice makes the question answerable: “adequate for which decision, under which conditions?” It also gives a reviewer a basis for requesting additional evidence instead of relying on a general trust label.

Alternatives. An exploratory notebook can use a lighter statement of purpose. A domain-specific assurance process may provide substantially stronger requirements. A single model rating is easier to display, but can hide differences among uses; it is not the default here.

Switch conditions. Increase formality when consequences, coupling, or external reliance increase. Use an applicable domain process where one governs the intended activity. Reduce administrative detail for disposable exploration while preserving the distinction between exploration and a supported decision.

Evidence and limits. The cited standard supports the use-specific principle. This publication does not show that jBOK's proposed record improves decisions or that any particular model is fit for use.

Evaluation needed. Give reviewers matched scenarios with and without the record, including a deliberately inappropriate reuse. Measure detection of the mismatch, review time, and unnecessary rejection. Revise the record if its added burden fails to improve those outcomes.

Choose and declare the reproducibility target

Origin: jBOK synthesis. Support: literature-supported foundations; proposed application.

Context. Debugging a simulator, comparing hardware, and estimating a stochastic policy's performance require different comparisons. “Reproducible” is incomplete unless the expected agreement and execution conditions are specified.

Recommended default. Declare whether the target is exact replay, agreement within justified numerical tolerances, or agreement of statistical conclusions. Record software and model versions, input identities, execution settings, random-number configuration, and relevant hardware. For stochastic claims, report variation across an appropriate set of runs as well as any replayable example.

Sources and adaptation. PyTorch's reproducibility documentation, introduction and randomness/algorithm sections describes seed control, nondeterministic operations, platform limitations, and performance tradeoffs. jBOK generalizes these practical concerns into an experiment-level declaration. PyTorch's guidance is implementation-specific; it is not a guarantee for other runtimes.

Rationale. An explicit comparison target helps distinguish a regression from an expected variation. Exact replay can make an anomaly easier to investigate without becoming a requirement that excludes useful stochastic methods.

Alternatives. Fully controlled execution is useful for debugging and some regression checks. Statistical comparisons suit variable outcomes. A tolerance check may be appropriate for numerical ports, provided the tolerance follows the quantity of interest rather than being chosen to make a test pass.

Switch conditions. Prefer tighter execution control while isolating a defect. Reassess the target when changing hardware, parallelism, solver, or precision. If exact control materially harms the intended workload, document that tradeoff and select a comparison capable of detecting consequential changes.

Evidence and limits. Seeded execution alone does not ensure identical outputs. Identical outputs do not establish physical accuracy. The proposed declaration has not been evaluated across independent jBOK implementations.

Evaluation needed. Introduce controlled changes to seeds, runtime versions, and execution settings. Test whether the declaration predicts which comparisons remain meaningful, and whether the chosen checks distinguish acceptable variation from a deliberately injected numerical defect.

Make parameter origins and missing declarations visible

Origin: jBOK synthesis. Support: literature-supported foundations; proposed application.

Context. A parameter's numerical value does not explain why it is credible. A measured quantity, fitted coefficient, operational setting, and convenient assumption may look identical in a configuration file while supporting different inferences.

Recommended default. Associate consequential parameters with their source, units, transformation history, responsible contributor, and role in the model. Use a practical vocabulary such as measured, derived, calibrated, configured, tuned, or assumed, allowing multiple stages in a lineage. Record missing declarations explicitly. Compare declared coverage with the components and values actually used in an execution.

Sources and adaptation. W3C PROV-DM (2013), §§2.1, 2.2.1, and 5.1–5.3 supplies entities, activities, derivations, and responsibility relationships, with extension mechanisms. jBOK proposes a simulation-specific vocabulary and execution-coverage check on top of those concepts. These categories and checks are jBOK choices, not requirements imposed by PROV.

Rationale. The intended benefit is a reviewer who can follow a disputed value back to its basis and identify what remains unknown. Linking records to execution would reduce the chance of reviewing documentation for a component that was never used.

Alternatives. A versioned table may be sufficient for a small model. A provenance graph becomes useful when derivations and shared dependencies are complex. Requiring a graph database for every experiment would add infrastructure before its benefit is demonstrated.

Switch conditions. Add structured relationships when manual tracing becomes unreliable, artifacts are reused across teams, or upstream changes need impact analysis. Keep the simplest representation that preserves the relationships required for the investigation.

Evidence and limits. PROV supplies a language for recording relationships. It cannot certify that a measurement occurred, a contributor's account is accurate, or a chosen parameter is suitable. Complete declarations can still describe a poor model.

Evaluation needed. Omit, replace, and misattribute selected inputs in a bounded example. Measure which discrepancies the proposed coverage check detects, which require human investigation, and what misleading declarations remain undetected.

Separate uncertainty statements from validity claims

Origin: jBOK synthesis. Support: literature-supported foundations; proposed application.

Context. An interval around a prediction answers a different question from whether the model represents the relevant physical process. Uncertainty propagation is conditional on the model and input information supplied to it.

Recommended default. State the quantity being predicted, relevant uncertainty sources and dependencies, propagation method, interval interpretation, and known omissions. Keep a separate description of the regime in which the model has been evaluated. Treat behavior outside that regime as a new question requiring evidence, even when the software still returns a number.

Sources and adaptation. JCGM 101:2008, §§5.1–5.5 and 5.9–5.11 describes propagation of input distributions, reporting, and conditions for Monte Carlo uncertainty evaluation in measurement models. jBOK adopts the discipline of explicit conditional statements. Extending it to evolving simulation and learned-model workflows requires additional treatment of model discrepancy and changing data; this source does not settle those problems.

Rationale. Separating these records should help prevent a precise interval from implying broader validity. It also gives future evaluation a concrete target: assess the uncertainty claim and the represented regime separately.

Alternatives. Analytic propagation can be efficient for suitable models. Sampling can handle cases where local approximation is inadequate, at added computational cost. Scenario analysis or bounds may better communicate uncertainty when a defensible probability distribution cannot be specified.

Switch conditions. Revisit the method when nonlinearity, dependence, tail behavior, or the intended decision makes the existing approximation inadequate. Preserve explicitly unresolved uncertainty rather than manufacture a probability distribution to satisfy a schema.

Evidence and limits. Numerical convergence of a sampling calculation does not establish the adequacy of its input distributions. No calibration result, validity envelope, or general uncertainty method is established by this entry.

Evaluation needed. Begin with analytically tractable cases, then use held-out observations with stated reference uncertainty. Assess interval behavior and error by regime. Include deliberate model mismatch to determine whether the reporting distinguishes numerical success from an inadequate representation.

Specify behavior as well as interface metadata

Origin: jBOK synthesis. Support: literature-supported foundations; proposed application.

Context. Two components can accept the same field names yet disagree about units, timing, state, or solver responsibilities. A well-formed message establishes only part of compatibility.

Recommended default. Separate descriptive metadata from the executable protocol. Specify initialization, state transitions, time advancement, event handling, units, error behavior, and required capabilities wherever they affect composition. Prefer established interfaces when they fit, then document the additional evidence and validity information a particular use requires.

Sources and adaptation. FMI 3.0.2, §§1.2, 2.3, and 2.4.3, with interface definitions in §§3–5 distinguishes Model Exchange, Co-Simulation, and Scheduled Execution and defines behavioral and descriptive elements. jBOK uses this distinction to constrain its own contract proposals. No implemented FMI mapping or compliance claim follows from this recommendation.

Rationale. The goal is to make component replacement a testable engineering claim. A reader should be able to identify which responsibilities remain with the orchestrator and which belong to the component.

Alternatives. A narrow local interface can be appropriate for a prototype. FMI may suit dynamic-model exchange, but does not automatically fit every agent or learned-model interaction. A bespoke protocol can serve unmet needs if its extra maintenance cost and intended interoperability boundary are explicit.

Switch conditions. Adopt a shared standard when independent tools need to cooperate and its semantics fit the problem. Retain a narrower interface when the standard adds substantial complexity without relevant capability. Revisit that decision when an external implementation needs to integrate.

Evidence and limits. Interface compatibility does not establish physical equivalence or stability of a coupled simulation. jBOK's draft contracts remain proposals; their existence is not evidence of an interchangeable ecosystem.

Evaluation needed. Have another developer implement a small component from the written contract alone. Exercise normal execution, boundary cases, and failures. Record every clarification needed, then test the composed system against a defined reference before claiming substitution succeeds.

Select fidelity and acceleration against an error budget

Origin: jBOK synthesis. Support: literature-supported foundations; proposed application.

Context. More detailed physics, higher arithmetic precision, and faster hardware address different limitations. A choice must account for the decision's accuracy needs as well as memory, throughput, latency, and cost.

Recommended default. Define the quantities of interest and acceptable error before optimizing. Establish a reference computation, then compare candidate discretizations, model approximations, precisions, and execution strategies against it. Record end-to-end cost alongside error and relevant failure cases. Preserve uncertainty about the reference itself.

Sources and adaptation. NVIDIA's Floating Point and IEEE 754 white paper, §§2.2–2.3, 5.3–5.4 explains how arithmetic order, fused operations, and implementation choices affect numerical comparisons. jBOK takes the general comparison discipline, not a vendor preference. The proposed decision-level error budget extends beyond that paper's floating-point scope.

Rationale. This makes acceleration useful when it delivers an acceptable result sooner or more economically. It also permits a simple model to remain the preferred option where extra complexity has no demonstrated decision benefit.

Alternatives. Higher precision, smaller timesteps, mesh refinement, reduced-order models, learned surrogates, and multifidelity estimation offer different tradeoffs. None is selected universally. An accelerated kernel may be irrelevant when data movement or another stage dominates the workload.

Switch conditions. Increase fidelity when observed error changes the decision. Consider cheaper approximations for repeated evaluations when their error is characterized in the required regime. Reassess precision and execution strategy when numerical sensitivity or hardware changes invalidate the existing comparison.

Evidence and limits. Hardware agreement is not physical validation, and a more expensive reference can share the same modeling error. This entry reports no speedup, hardware ranking, surrogate accuracy, or demonstrated multifidelity advantage.

Evaluation needed. Publish a bounded workload with reference justification, error measures, workload size, timing boundaries, and hardware/software versions. Compare alternatives at matched error targets; include setup and transfer costs where relevant. Report cases in which acceleration or added fidelity fails to help.

How these entries will change

Each recommendation should become narrower or better supported as comparisons and implementation experience accumulate. Revisions should identify the changed judgment and its reason, preserving a distinction between a literature update, a private observation, and publicly reproducible evidence. The research agenda describes the next evaluations; the draft contracts explore how selected ideas might be represented in software.