# Test plan: does a large CTR gap pick the cheaper conversation?

Fixed 8 November 2026, before any advertising data dated after 25 October 2026 has been read for this study. Responsible author: Ooi Yik Jinn. This is a plan only: running it needs the author's go-ahead and a read-only Meta API pull. Nothing here authorises a data read, a publication or a change to any ad.

*Publication copy, 9 November 2026: two attribution sentences are reworded. The rules, thresholds and Change 1 are as fixed on 8 November 2026.*

## Why

Version 0.8 reports an exploratory, post-hoc pattern in the January–October 2026 archive. When the higher-CTR ad's link CTR was at least 1.5 times its alternative's, it had the cheaper recorded conversation in 26 of 32 comparisons. Below 1.5 times it did so in 15 of 29. The cut-point was chosen as a round number before that result was seen, but after the study's other results were known. The pattern needs data it has never touched.

## Hypothesis

Using the unchanged primary rules, agreement between higher link CTR and lower cost per reported conversation is **higher when the CTR ratio is at least 1.5 than when it is below 1.5**.

## Data

- Complete Monday–Sunday weeks from **26 October to 20 December 2026** (eight weeks).
- Pull read-only, **no earlier than 3 January 2027**, so recorded actions can settle. Use the Meta Marketing API version current at the pull and record it. If the API version differs from v26.0, record any field changes before analysis.
- Same account approach as `validation-protocol.md`: attempt the archive's account list, save access failures, and associate identities using the approved register current at the pull date.
- Same eligibility, matching, metric and missing-action rules as `protocol.md`: MYR, `REPLIES`, WhatsApp-only, Malaysia-only targeting, recruitment excluded, same account/ad set/week/objective/attribution setting/destination, at least three shared positive-impression dates, and RM30 and 1,000 impressions per ad.
- CTR ratio = higher link CTR ÷ lower link CTR, exact arithmetic. Exact CTR ties and both-zero comparisons are excluded, as before.

## Sample-size rule, decided before any outcome

Before computing any CTR or conversation figure, count the **qualifying** pair/weeks (exposure floors only). If fewer than 30 qualify, extend the window in four-week blocks (to 17 January 2027, then to 14 February 2027), each pulled at least 14 days after its end. Stop at 14 February 2027 whatever the count. Record each count and extension decision with its date.

## Primary analysis

- Estimand: **D = agreement among comparisons with ratio ≥ 1.5 minus agreement among those with ratio < 1.5**, over all rankable pair/weeks in the test window only. Never pool the test window with the January–October archive for this claim.
- Uncertainty: percentile bootstrap resampling whole distinct ad pairs, 20,000 replicates, `random.Random(20270101)`. Report the 2.5th and 97.5th percentiles.
- Verdict, fixed now (**superseded by Change 1 below**, kept for the record):
  - **Supported**: D > 0 and the lower bound of the pair-clustered interval is above 0.
  - **Contradicted**: D ≤ 0.
  - **Inconclusive**: D > 0 but the interval includes 0, or either group has fewer than 8 rankable comparisons.
- Expect "inconclusive" to be a real possibility: eight weeks gave only 20 rankable comparisons in the later 2026 period.

## Secondary analyses, all reported whatever they show

1. Agreement in the ≥ 1.5 group alone, with its pair-clustered interval.
2. The same split at the earliest rankable week per distinct pair within the test window.
3. The same split restricted to pairs **not** among the 30 pairs in the January–October analysis.
4. An owner-clustered bootstrap of D (seed 20270102). The owner count stays private.
5. Agreement among one-unit-robust and one-unit-fragile comparisons (descriptive, outcome-conditioned).

## What this test cannot show

It cannot show a causal effect of raising CTR, a national rate, lead quality, sales, or that a CTR-based switching rule beats a cost-based one. It tests one descriptive pattern on new weeks from the same kind of selected archive.

## Changes

Any deviation from this plan is written below, dated and explained, **before** the test-window results are computed.

### Change 1 — 8 November 2026, before any test-window data was read

**Why.** A review of the draft found the original verdict rule unsound:
"Supported" and "Inconclusive" could both apply when a group had fewer than eight comparisons; "Contradicted" needed only
D ≤ 0, with no uncertainty, while "Supported" needed an interval clear of zero; and the bootstrap silently discarded
resamples that lacked one of the two groups, which can produce a spuriously certain interval when few pairs carry a group.

**Replacement verdict rule.** Apply these steps in order; the first that applies decides, so the verdicts are mutually exclusive.

1. **Support check.** If either group (CTR ratio below 1.5, or 1.5 or more) has fewer than **8 rankable comparisons** or
   fewer than **5 distinct ad pairs**, the verdict is **Inconclusive (insufficient support)**.
2. **Resampling check.** Run the pair-clustered bootstrap (20,000 replicates, `random.Random(20270101)`). Count every
   replicate. A replicate in which either group is empty is **undefined** and is kept in the count, not discarded. If more
   than **1%** of replicates are undefined, the verdict is **Inconclusive (unstable resampling)**. Otherwise the interval is
   the 2.5th and 97.5th percentiles of the defined replicates.
3. **Supported** if the interval's lower bound is above 0.
4. **Contradicted** if the interval's upper bound is below 0.
5. Otherwise **Inconclusive**. The sign of D is then reported as a description, not as evidence either way.

Report the number of undefined replicates and the distinct-pair count in each group whatever the verdict.

**Limit added.** The sample-size rule is outcome-blind in its calculation (it counts exposure-qualifying pair/weeks before
any CTR or cost figure is computed). It is not statistically independent of performance: whether ads keep receiving delivery,
and whether action lists are returned, can depend on how they perform. Thirty qualifying pair/weeks also need not give thirty
rankable comparisons or balanced groups; the support check above handles that case.
