Diagnostics Gates

For the workflow interpretation of these checks, start with Stage 4: Computation and Sampler and Stage 5: Model Adequacy. This page is the threshold and policy reference.

Use this page when you need exact gate thresholds, status aggregation, or YAML policy semantics. It does not replace substantive model review: a model can clear threshold tables and still be a poor basis for interpretation.

Geographic FE/RE/CRE models also have separate pre-fit panel design diagnostics, including same-date geographic contrasts and conditional design support.

Editable gate profiles

Edit config/diagnostic_gates.yml to define team reporting thresholds. The shipped BLM, RE and CRE examples reference this complete profile:

diagnostics:
  enabled: true
  gates_file: diagnostic_gates.yml

Relative paths resolve from the model YAML directory, including when runme.R applies CLI data or holiday overrides. The file is validated before compilation, including --mode=validate. Each profile requires schema_version: 1, a non-empty profile name and description, and the complete thresholds mapping. For example, one entry in that mapping is:

thresholds:
  design_max_vif: {warn: 10, fail: 30}

The profile controls reporting, not the likelihood, priors, numerical design checks or estimator support. Strict > and < comparisons are used; equality does not breach a threshold, and failure takes precedence. Equal warn/fail thresholds have no warning band. fail: null makes a finite result warning-only. Non-finite evidence for an evaluated check cannot pass; checks that do not apply remain skipped with reasons. MCMC gates are skipped for MAP. Empty or entirely skipped evidence is not PASS and has unknown publishability.

Unknown or duplicate keys, incomplete mappings, non-finite thresholds, invalid domains and reversed warn/fail ordering are errors. warn is required and numeric; fail must be explicitly numeric or null. Do not combine a gate file with inline pareto_k_warn/pareto_k_fail or abs_corr_warn/abs_corr_fail.

An explicit file controls all configurable gate classifications, including failures in publish mode. Without a file, the package resolves the existing mode and inline thresholds into a named default profile. policy_mode still controls existing structural/preflight policies that are outside this file. A gate file cannot relax estimator-specific rank or support requirements.

The runner freezes the resolved profile before fitting and records its source, content hash and hash algorithm in 40_diagnostics/diagnostic_gates.resolved.yaml. The hash uses rlang::hash; it is labelled rlang_hash, not SHA-256. Editing a profile during sampling does not change the active run. Version-control the profile with the model configuration; change its name when issuing a new team policy. The resolved snapshot contains provenance fields and is an output artifact; use the four authored fields above when creating a new input profile.

VIF interpretation

VIF is a collinearity diagnostic, not a hypothesis test. The default warning above 10 and failure above 30 reuse the existing preflight thresholds. The report records the maximum and limiting term; diagnostic_vif.csv retains per-term values, statuses, limits and design scope. vif_bar.png uses the same resolved rule. Fewer than two population predictors is not applicable; singular or constant predictors cannot produce a passing VIF gate.

This population-design VIF is distinct from CRE between-design VIF in cre_estimability_vif.csv. Those estimator-specific diagnostics and safeguards remain unchanged. The standard diagnostic-report API remains unsupported for FE; FE runs retain their existing dedicated estimator artifacts. FE configs reject diagnostics.gates_file and do not produce a standard gate snapshot. No generic FE gate packet or new estimator qualification is implied.

Terminal output

Rscript runme.R and the package CLI print grouped diagnostic results, observed values, threshold rules, limiting terms/parameters/chains where available, skipped reasons and recovery guidance. The report is derived from the same structured checks saved to CSV. Colours supplement explicit PASS/WARN/FAIL/ SKIPPED words; plain and redirected output remain readable. A diagnostic FAIL is distinct from a failed fit: retain and inspect the fitted model and reports. Existing CLI codes remain 0 for completion without diagnostic failure, 1 for completed runs needing failure review, and 2 for execution/configuration errors.

Model selection for time-ordered data

PSIS-LOO and WAIC treat pointwise observations as conditionally exchangeable. That assumption is not generally appropriate for time-ordered MMM data, where nearby weeks can remain dependent after conditioning on the fitted model.

Use the runner’s expanding-window blocked CV or leave-future-out CV as the primary evidence when selecting among time-series MMM specifications. Treat PSIS-LOO, WAIC, and Pareto-k outputs as supplementary fit and influence diagnostics. They do not establish future-period predictive performance, causal validity, or a publish-gate pass on their own.

Purpose

dsambayesplus runs a deterministic diagnostics framework after model fitting. Each diagnostic check produces a pass, warn, or fail status. The resolved gate profile controls the configurable thresholds. This page defines the check taxonomy, threshold tables, policy modes, identifiability gate, and the overall status aggregation rule.

How to use this page

• Use Stage 4: Computation and Sampler to understand which checks are non-negotiable before trusting the posterior.
• Use this page to see the exact dsambayesplus thresholds and artifact semantics.
• Use Stage 5: Model Adequacy before treating a passing diagnostics table as permission for decomposition, model comparison, or optimisation.

Policy modes

Without an explicit gate file, the diagnostics framework resolves these preset modes from diagnostics.policy_mode:

The mode is resolved by diagnostics_policy_thresholds(mode) in R/diagnostics_thresholds.R.

Check taxonomy

Checks are organised into phases:

Each check row includes:

Design checks

Condition number thresholds by mode

P0 sampler checks (MCMC only)

Mode adjustments for sampler checks

dsambayesplus treats any Rhat above 1.01 as a failure in publish and strict modes, following the rank-normalised, folded Rhat guidance in Vehtari et al. (2021) and the Stan warnings guide. In explore mode, the fail threshold is deliberately relaxed to 1.10, while the warning threshold remains 1.01.

P1 residual checks

Mode adjustments for residual checks

In publish mode, values beyond the raw residual fail boundary remain warn. The persisted threshold field states that failure is disabled and retains the raw boundary for triage. Non-residual P0 failures can still make the overall run fail.

P1 boundary hit check

In explore mode, boundary hits cannot fail. Publish mode retains the raw fail > 0.20 boundary for triage but records values beyond it as warn. Failure is disabled for this check in publish mode. In strict mode, thresholds tighten to warn > 0.02, fail > 0.10.

P1 within-group variation check

This check applies to hierarchical models and flags groups where within-group variation is extremely low relative to between-group variation. In explore mode, the fail threshold is zero (cannot fail). Publish mode retains the raw fail < 0.05 boundary for triage but records values below it as warn; failure is disabled for this check. Strict mode applies warn < 0.15 and fail < 0.10.

Identifiability gate

The P1 pre_identifiability_baseline_media_corr check measures the maximum absolute correlation between baseline terms and media terms in the design matrix. It is configured via diagnostics.identifiability in YAML:

diagnostics:
  identifiability:
    enabled: true
    media_terms: [m_tv, m_search, m_social]
    baseline_terms: [trend, seasonality]
    baseline_regex: ["^h_", "^sin", "^cos"]
    abs_corr_warn: 0.80
    abs_corr_fail: 0.95

Term detection

• Media terms: explicitly listed in media_terms.
• Baseline terms: union of baseline_terms, generated time-component terms, and matches from baseline_regex patterns.
• Both sets are intersected with actual design-matrix columns and filtered to remove constant columns.

Thresholds by mode

Skip conditions

The identifiability gate reports skipped when:

• identifiability.enabled: false
• No configured media terms found in the design matrix
• No baseline terms detected from configured terms/regex
• All resolved baseline or media terms are constant

Overall status aggregation

The overall diagnostics status is determined by diagnostics_overall_status():

1. If any check has status == "fail" → overall status is fail.
2. If any check has status == "warn" (and none fail) → overall status is warn.
3. Otherwise → overall status is pass.

Checks with status == "skipped" do not affect the overall status.

P2 rows use the emitted post_model_selection_method, post_psis_loo, post_psis_loo_options_ignored, post_psis_loo_time_dependence, post_psis_loo_elpd, post_psis_loo_looic and post_psis_loo_pareto_k identifiers. They record supplementary predictive evidence and do not override P0 or P1 failures.

The runner also emits post_sampler_params_available when sampler parameters can be inspected and post_sampler_unavailable_for_optimise for the explicit MAP not-applicable path. These identifiers are part of the persisted report; they are not substitute MCMC diagnostics for a MAP fit.

Runner artefact output

The diagnostics framework produces:

Interpretation guidance

• skipped: no evaluated evidence; this is not a gate pass.
• pass: no enabled check breached its configured threshold. This does not establish substantive adequacy, causal validity or suitability for the intended decision.
• warn: review recommended; the model may have quality concerns but does not block the configured policy.
• fail: at least one enabled check breached a fail threshold; resolve it before using the result for decisions.

Common remediation actions

Cross-references

• Model Classes, which diagnostics apply to each class
• Response Scale Semantics, residuals are computed on model scale
• Config Schema, diagnostics.* YAML keys
• Output Artefacts, diagnostics artefact paths
• Diagnostics Plots, visual diagnostic outputs