# Cartographer simulation model Engine version: **1.0.0** This document specifies the estimand, data-generating process, evidence boundary, strategy approximations, and reported metrics in `engine.js`. The simulation is a falsifiable methods illustration, not a calibrated forecast of any company. ## Target estimand For lever \(i\), segment \(s\), and the other levers \(X_{-i}\), the signed effect is the fixed-range average marginal effect \[ \theta_i = \sum_s w_s\;E_{X_{-i}}\left[\frac{f(X_i=0.8,X_{-i},s)-f(X_i=0.2,X_{-i},s)}{0.6}\right]. \] The segment weights are fixed at \((0.46,0.34,0.20)\). The context expectation uses 24 deterministic, seeded draws. The discovery target is sensitivity \(|\theta_i|\), so a lever with a large harmful intervention effect is still highly sensitive. The task is to identify the `topK` largest absolute effects. This is an average causal contrast over a declared interval and context distribution. It does not measure recovery of the complete response surface. `mapRMSE` therefore compares estimated lever sensitivities with \(|\theta_i|\), not predictions of \(f(X)\) everywhere. ## Hidden response surface For segment \(s\), the outcome without observation noise is \[ f(X,s)=\sum_i \beta_{is}g_i(X_i)+ \sum_{(i,j)\in E}\gamma_{ijs}(X_i-0.35)(X_j-0.35). \] The seeded generator assigns each lever a linear, diminishing-return, threshold, inverted-U, step, or negligible shape. Segment coefficients vary and can reverse sign. The number and magnitude of pairwise interactions increase with `complexity`. The context generator combines a shared uniform draw within groups of four levers with an independent uniform draw: \[ X_i = 0.28U_{group(i)}+0.72U_i. \] The engine retains an actual `response(X, segment)` function and uses it to generate historical outcomes. It computes the displayed estimand with the algebraically equivalent sparse finite-difference expression. For the bilinear interaction above, the marginal interaction contribution is \(\gamma_{ijs}(X_j-0.35)\). This avoids repeatedly evaluating the full response for every lever, segment, and context. `effectSparsity` controls the proportion assigned to the nominal strong-effect tier; its default of 1/6 gives five strong levers when there are 30. One third of levers are nominally medium and the rest weak. These tiers set coefficient distributions before shape, heterogeneity, and interactions. They do not force the nominal strong levers to be the realized top five. Historical rows draw segment, context, a latent confounder, and outcome noise. The latent variable shifts several observed lever values and also shifts the outcome, with magnitude controlled by `confounding`: \[ Y=f(X,s)+0.9cZ+0.45\,\text{noise}\,\epsilon. \] The visible opportunity evidence contains reach, drop-off, an impact proxy, confidence, and their RIC-style product. These are constructed from historical marginal associations, benchmark gaps, complaint-like signals, and sample size. To guarantee the specified stress case, the generator chooses a lever from the bottom sensitivity quintile and gives it conspicuously high opportunity signals. The policy sees those signals but never receives the evaluator's `misleading` annotation. ## Evidence boundary All four strategies start with the same evidence rows and the same weak initial belief. Policy state contains configuration, the shared evidence summaries, current beliefs, action counts, and assumed source variances. It does not contain the landscape, true effects, evaluator rankings, or evidence-source centers. The evidence oracle is the only component that can access a hidden causal effect. It uses that access only to generate micro-test and RCT outcomes. Historical evidence comes from sampled outcomes of the hidden response function. Offline estimates and synthetic estimates are fitted from permitted historical summaries. The synthetic center is calculated from an offline historical fit and standardized RIC evidence: \[ m_i^{data}=0.82m_i^{offline}+0.18\,SD(\hat\beta)z(RIC_i), \] \[ m_i^{syn}=q\,m_i^{data}+(1-q)m_i^{prior}+b_i, \] where \(q\) is the configured `fidelity` and \(b_i\) is a fixed seeded model error whose scale depends on `syntheticBias` and \(1-q\). Neither term reads \(\theta_i\). Synthetic random error also decreases with the fidelity setting. Consequently, `fidelity` is a target reliability setting in the proxy error model, not a claim that realized accuracy equals the slider value. Every result reports realized sign accuracy, pairwise sensitivity-rank accuracy, top-k recall, and proxy/truth correlation separately. A particular finite run can have lower realized accuracy at a higher target setting. ## Approximate fits and belief updates The offline fit is deliberately inexpensive. It combines marginal and segment-specific historical slopes with shrinkage controlled by `priorQuality`; it is not a full multivariate causal regression. This makes the browser simulation fast and preserves vulnerability to confounding, omitted interactions, and nonlinear misspecification. Beliefs use Gaussian summaries. Each source observation has random standard deviation \(\sigma\) and a non-shrinking systematic-bias floor \(b\). For \(r\) repeated observations in a source family, its effective variance is \[ V_{family}=b^2+\left(\sum_{j=1}^{r}\sigma_j^{-2}\right)^{-1}. \] Historical analysis, offline fitting, and synthetic probes share one correlated `model` family because they reuse the same history. Micro-tests and RCTs are separate families. Repeated cheap queries reduce random error but cannot remove the shared floor. The displayed map `sd` is the posterior standard error under this approximate Gaussian model; it is not the raw observation SD or a guarantee of frequentist coverage. ## Strategies All strategies have the same maximum budget and cannot overdraft it. They may finish below the maximum after exhausting their permitted action schedule. - **Opportunity Sizing** ranks by reach × estimated impact × confidence, validates in that order with RCTs, reranks using observed effects, and uses remaining budget for uncertainty-focused repeat validation after every lever has been covered. - **Offline Analyst** pays the configured offline-analysis cost for each lever, then runs micro-tests in estimated-effect and uncertainty order. If budget remains after coverage, it repeats micro-tests near the selection boundary. - **Synthetic Explorer** pays for offline fits, adds one synthetic probe per lever, micro-tests the leading candidates, and uses RCTs for the remaining leading candidates. - **Cartographer** considers analysis, synthetic probe, micro-test, and RCT actions sequentially. For a proposed action, it calculates the incremental family precision \(\Delta\tau\) and approximate information gain \[ I(a)=\frac12\log(1+s_i^2\Delta\tau_a). \] It weights this by proximity to the current top-k boundary, \[ w_i=0.14+\exp\left(-\frac{||m_i|-c_k|}{\max(0.05,s_i+0.08\,scale)}\right), \] and selects the available action maximizing \(w_iI(a)/cost(a)\). This is a tractable, top-k-focused approximation to expected information gain, not exact mutual information over every possible top-k set. Per-lever action caps prevent vanishingly small numerical gains from creating unbounded rounds. There is no hidden performance bonus in the acquisition score. ## Costs and exposure Default learning-unit costs are analysis 1, synthetic 2, offline 5, micro-test 25, and RCT 100. `validationCost` multiplies micro-test and RCT cost. A micro-test records 120 exposure units and an RCT 900; offline and synthetic work record zero. These exposure units are illustrative and are not customers, dollars, or elapsed days without an external calibration. Let `T` be the standard deviation of the observed historical slopes, floored at 0.15; `v` the noise setting; `c` confounding; `h` complexity; `q` proxy fidelity; and `b` synthetic bias. Assumed source error parameters, in outcome units, are: - Analysis: random SD `T(0.42 + 0.35v)`; bias floor `T(0.18 + 0.55c + 0.22h)`. - Offline fit: 0.78 times the analysis random SD, with the same floor. - Synthetic probe: random SD `T(0.35 + 1.2(1-q))(0.7 + 0.3v)`; floor `T(0.10 + 0.75b + 0.55(1-q))`. - Micro-test: random SD `T(0.20 + 0.30v)`; assumed floor `0.08T`. Its actual persistent shift is `T(0.045 + 0.10h)` times a seeded standard-normal draw. - RCT: random SD `T(0.07 + 0.13v)`; assumed floor `0.025T`. Its actual persistent shift is `0.018T` times a seeded standard-normal draw. The noise control adjusts part of the error budget; zero does not remove every source's residual error. Initial history consists of observational outcomes, not a bank of randomized experiments. Live-test actions are Gaussian summary observations of the declared average contrast, not simulated individual customer randomization. Both are simplifying assumptions of this version. ## Evaluation - `recall`: selected true-top-k levers divided by k. - `precision`: selected true-top-k levers divided by k. Because selection size is fixed at k, precision equals recall. - `cost`: learning units spent by the endpoint. - `exposure`: illustrative live-validation exposure units. - `rounds`: evidence actions taken. - `mapRMSE`: root mean squared error of the estimated absolute sensitivity map. - `regret`: normalized top-k sensitivity-set regret, \((\sum top\ k\ sensitivities-\sum selected\ sensitivities)/\sum top\ k\ sensitivities\). This is not joint intervention outcome regret or foregone revenue: interactions can make a set's joint value differ from the sum of its marginal effects. - `discovered`: whether the strategy ever selected the complete top-k set. - `discoveryCost`: first cost at which that occurred, or `null` if it never occurred. - `finalRecovered`: whether the strategy retained the complete set at its endpoint. Monte Carlo `recoveryRate` is the uncensored proportion ever recovering all k levers. `finalRecoveryRate` is the endpoint proportion. Discovery-cost summaries contain recovered runs only and report their `n`; failed runs are not assigned the maximum budget as a fabricated discovery cost. Monte Carlo intervals are normal 95% confidence intervals for means. Paired differences use within-landscape differences because every strategy sees the same generated business and initial evidence. A one-run summary has no inferential interval (`lo`, `hi`, and `sd` are null). The default 1,000-landscape run took about six seconds in Node on the development Mac after sparse estimand optimization. Browser hardware varies, so the UI runs it in a worker and may chunk sweeps for progress. The interface and batch runner also compute Wilson 95% intervals for endpoint and ever-recovery proportions. First recovery is an evaluator-only event, not a stopping certificate available to a policy. All strategies continue their own action schedules after such an event, and can later lose the correct selection. A common base seed across parameter settings aids reproducibility; changing dimensionality does not preserve an identical underlying landscape. Within each setting, all strategy comparisons use paired landscapes. ## Limitations The response functions, evidence costs, exposure counts, segment weights, and error distributions are stylized. Marginal historical fits can be badly biased under confounding and interaction. Gaussian belief updates do not fully represent multimodal top-k uncertainty, and the Cartographer acquisition rule approximates rather than exactly integrates top-set entropy. The deliberate misleading opportunity is a stress condition, not an estimate of its real-world frequency. Normal Monte Carlo intervals can be imperfect near bounded metric endpoints. Results support comparisons inside the declared simulation regime only. Method design was informed by the ADEMP simulation-study framework in Morris, White, and Crowther (2019), cost-sensitive information acquisition in Song, Chen, and Yue (2019), and fixed-confidence multiple-arm identification in Kaufmann, Cappé, and Garivier (2016): - Morris, White & Crowther, [Using simulation studies to evaluate statistical methods](https://arxiv.org/abs/1712.03198) - Song, Chen & Yue, [A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes](https://proceedings.mlr.press/v89/song19b.html) - Kaufmann, Cappé & Garivier, [On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models](https://jmlr.org/papers/v17/kaufman16a.html) These references motivate the evaluation structure and acquisition posture; the engine does not claim to reproduce their algorithms.