A portfolio company reports that its strategy workshop produced an outbound experiment: 40 target accounts, four signed pilots and a €1,500 spend cap. At review, three pilots have signed. Whether that is encouraging depends on who was contacted, how long each account was observed, what the pilots require and whether the original rule survived the test.
Review the chain from assignment to execution, measurement and inference before discussing the next commitment. Teams Facilitator's reviewed canvas and summary preserve the intended choice; company operating records establish what actually happened.
Recover the rule recorded before results arrived
Request the dated hypothesis, population, offer, resource limits, outcome definition and continue/stop rule. Strategyzer's Test Card separates hypothesis, experiment, measure and threshold. Add delivery constraints and the person authorized to release further resources.
Compare that original record with the current explanation. “Four pilots” may have become “three serious conversations”; the target buyer may have broadened after poor response. Changes can be sensible, but date them and treat the changed test separately.
Use this minimum review table:
| Layer | Review question | Evidence |
|---|---|---|
| Assignment | Which accounts met the original eligibility rule? | Dated target list and exclusions |
| Exposure | Which received the planned offer and sequence? | Outreach records |
| Outcome | Which signed the defined pilot after sufficient follow-up? | Agreements and cohort dates |
| Economics | What acquisition and delivery resources were consumed? | Spend, labor and scope records |
| Decision | Which original condition passed, failed or remains unresolved? | Rule compared with the evidence |
Resolve incomplete execution before judging the hypothesis
Suppose only 25 of the 40 accounts received outreach. Report the contacted cohort and the 15 unexposed accounts separately. If the founder-hour cap was exhausted, the experiment also revealed that the proposed acquisition process may be too labor-intensive.
Do not silently replace “per eligible account assigned” with “per account contacted.” Those answer different questions. The first includes execution feasibility; the second describes response among accounts actually reached, potentially a selected subset.
Ask why accounts were skipped. If difficult prospects were excluded late, the observed rate may overstate what the original target population would produce. A completed canvas cannot establish that this selection was harmless.
Give each sales cohort the promised observation window
Accounts contacted yesterday have had less opportunity to sign than accounts contacted a month ago. Combining them in one conversion rate treats unfinished observation as failure.
In the fictional experiment, suppose 20 accounts have completed the agreed 30-day follow-up and produced three pilots. Another 20 are only ten days into that window. Report three of 20 for the mature cohort, and mark the later cohort as pending. Preserve all 40 in the execution report.
Early successes can be counted, but comparing a mature cohort with a partially observed one requires care: the quickest conversions arrive first. Schedule the final review from the last eligible exposure, or use an appropriate time-to-event analysis when the decision justifies that complexity.
Also inspect late cancellations and delivery acceptance. A signature may be an appropriate acquisition event while remaining an incomplete measure of usable revenue.
Interpret zero outcomes with the denominator attached
Zero pilots from ten fully observed, independent comparable accounts does not establish zero demand. Hanley and Lippman-Hand's original zero-event analysis explains the uncertainty hidden by a zero numerator.
For zero events in n independent Bernoulli trials with a common probability, the exact one-sided 95% upper confidence limit is 1 − 0.05^(1/n). At n = 10 it is about 26%; at n = 40, about 7.2%. The familiar 3/n approximation is more useful at larger sample sizes.
This applies only to the probability modeled. Shared reseller introductions, changing offers and unequal follow-up undermine the simple independent-trial interpretation. Even with clean execution, zero of 40 does not rule out a commercially attractive 5% conversion rate. It may still justify stopping because the resource cap was reached; a management limit and a statistical conclusion are different claims.
Separate a commercial signal from causal lift
Four acquired customers establish observed acquisition under the tested conditions. They do not show how many would have purchased without the campaign. Founder relationships, concurrent launches or seasonal demand may explain part of the outcome.
Microsoft's controlled-experiment guidance explains why randomization and proper design support causal inference. Where feasible, use a suitable concurrent randomized comparison with enough observations. A small founder-led sales test may instead provide feasibility evidence; label its inference accordingly.
Avoid comparing channels tested against different buyers, offers or maturity windows and declaring the highest raw rate the winner. In a Bullseye review, examine those differences before reallocating the whole budget.
When many tactics are tried, the best observed result can partly reflect favorable noise. Preserve how many tests were considered, including abandoned ones. Before a large commitment, consider a fresh bounded replication of the leading tactic using the same eligibility and offer. That checks repeatability without pretending the first small sample precisely estimated future performance. Include replication cost in the next authorization rather than treating it as permission for unrestricted rollout.
Check measurement integrity before celebrating a pass
Microsoft's experiment-analysis checklists emphasize design and data checks across the experiment lifecycle. Apply that discipline to duplicated accounts, missing outcomes, revised eligibility and inconsistent attribution.
Check delivery economics separately. If four pilots sign only after bespoke commitments, the standard-offer hypothesis failed even when the acquisition count passed. Bring that dependency into Business Model Canvas rather than scaling an unpriced service obligation.
Keep repeated statistical inspection distinct from operational monitoring. Stop immediately when a predefined harm or resource limit is breached. If the company wants significance-based stopping whenever results look favorable, it needs an analysis designed for sequential decisions; ordinary fixed-horizon claims do not automatically remain valid.
Allocate the next intervention to the identified failure
Missing exposure calls for execution redesign. Immature observations call for follow-up time. A broken measure calls for repaired measurement. Adverse evidence under a completed test calls for a changed hypothesis. A passed demand measure with failed delivery economics calls for a model decision.
Record that diagnosis in the fund's existing tracker with the next owner and trigger. Retain the original session record so future reviews can distinguish new learning from rewritten history. Report unresolved evidence explicitly instead of forcing every company into a green or red status.
Run the first session with this review contract in place. The portfolio rollout playbook standardizes the handoff, and the founder-led guide helps companies choose what to commit while important information remains uncertain.