When to distrust marketing mix model results

Key Takeaways

• • Any marketing mix model will return a number if you ask it to, and whether that number is evidence is a separate question.
  • Four failures deserve refusal: the arithmetic does not close, the baseline cannot be identified, media gets almost no credit, or the sampler has not settled.
  • Some unusual results should publish with a caveat instead, because the model measured something and the answer is simply unexpected.
  • A refusal that names its cause and its remedy is more useful than a confident figure you cannot check.

Knowing when to distrust marketing mix model results is easier when the software says so itself. Most modeling tools treat an answer as the goal, whereas I now believe that recognizing when to withhold one is more valuable. A refused fit costs you a number today, and a published bad fit can cost you a budget decision next quarter.

This post makes the case for refusal, lists the checks I think justify it, and separates them from the results that deserve a warning instead. If you prefer a broader map first, the guide to validating a marketing mix model ties the series together.

What it means for a model to refuse

A refusal means the software declines to publish a fit, and explains why, because its own checks show that the numbers would not constitute evidence. It differs from an error message, since the model ran to completion and the decision to withhold comes afterward, from tests on the finished result.

The idea is not exotic, since Google's Meridian documentation includes health checks that return pass, review or fail statuses for convergence and negative baselines. A fail is a defined outcome there rather than an accident.

A refusal also has a second half that people overlook: the message must be specific enough to act on. "Fit failed" teaches nothing. "Your channels moved together too closely to be separated, so vary budgets independently for a few weeks" points at a fix.

Why a confident number can be worse than no number

A number invites action, and the consequences of that action are where the cost accumulates. If a fit is wrong but looks precise, a team may move budget on it, whereas a withheld fit leaves the plan intact.

Published research supports the concern. In 2017, Chan and Perry at Google described the obstacles that keep media mix models from delivering valid answers consistently. The spread around an answer matters as much as the answer, as I explain in reading a credible interval in a marketing mix model. A model that hides the spread, or publishes when the spread is unusable, gives a false sense of settled knowledge.

Knowing when to distrust marketing mix model results: four blocking failures

Four failures justify blocking a fit outright, in my view. The arithmetic does not close, the data cannot pin down the baseline, media gets almost no credit, or the sampler has not settled. In each case the model has not measured the thing you asked about, so no figure is worth caveating.

Figure 1. The four refusal checks, as implemented in Bayesin. Any one failure refuses the fit, and the page states the reason.

The decomposition must close

The model splits revenue into a baseline and media contributions. If the media contributions add up to more than the revenue you booked, the split does not close. A baseline share below zero triggers refusal with that explanation. The usual cause is spend that barely varies, so the model cannot tell a steady channel from the baseline. I cover the mechanics in what a baseline share too high or too low means.

The baseline must be identifiable

Baseline share is the part of revenue the fit does not attribute to media. The platform also refuses when the 10th to 90th percentile range of that share includes zero. It refuses too when more than 5% of the posterior sits below zero. A range spanning zero means the data does not answer the question. Meridian's health checks address a related concern by treating a consistently negative baseline as a clear problem.

Media must get some credit

If the fit credits media with under 5% of revenue, the platform refuses it. Either media truly moves nothing, or the channels rise and fall together so closely that they cannot be told apart. The fit cannot distinguish these two explanations, and publishing it would present an ambiguity as a finding.

The sampler must have settled

A Bayesian fit runs several chains that must agree. The platform requires a largest R-hat below 1.01, no divergences, and effective sample sizes of at least 100 per chain. It applies the same test to the second fit used for its accuracy check. Stan's guidance says to trust a sample fully only when R-hat is below 1.01. It adds that divergences after warmup should not be ignored if you want reliable inference, and MMM convergence and R-hat for non-statisticians unpacks these diagnostics.

Refuse or warn: where the line falls

A model should warn rather than refuse when it measured something and the result is merely unusual. A business can genuinely purchase most of its demand through advertising, and withholding that finding would be an error of its own. The line I draw separates a fit that did not measure the target from a fit that measured an unexpected one.

• Baseline share below 0% — Bayesin's action: Refuse; Reason: The split does not close
• Baseline range includes zero — Bayesin's action: Refuse; Reason: The data does not answer the question
• Baseline share above 95% — Bayesin's action: Refuse; Reason: Media gets almost no credit
• Sampler unsettled — Bayesin's action: Refuse; Reason: The posterior cannot be trusted
• Baseline share under 20%, range clear of zero — Bayesin's action: Warn; Reason: Measured, but unusual
• Baseline share from 85% to 95% — Bayesin's action: Warn; Reason: Channels may be inseparable
• History of 60 weeks or fewer — Bayesin's action: Warn; Reason: Too short to hold back 8 weeks for an accuracy check
• History under 52 weeks — Bayesin's action: Block; Reason: Seasonality cannot be separated from media

The 20% and 85% marks come from Measured's published QA guidance, which I describe in full in the baseline share post. A flag is an instruction to question the result, and it does not mean that the result is impossible.

How a refusal should read to the person who receives it

A good refusal states the problem in its first sentence and then gives a test the reader can run on their own business. The reader should not need to know what R-hat means. The supporting statistics can sit at the end, in parentheses, for anyone who wants to verify them. The example below is illustrative, with synthetic numbers, written in the style the platform uses.

Illustrative refusal message. The model credits marketing with more than the whole of the outcome: 1.30 million against 1.10 million actually recorded. Every business has demand that arrives without marketing, so a decomposition that leaves none is arithmetic that does not close, not a finding. The usual cause is spend that never varies much across the uploaded weeks. Check whether any channel dropped near zero, and vary budgets independently for several weeks before running again. (Baseline share: -18%.)

That ordering matters more than it seems. If a caveat read "media accounts for 99.2% of revenue," a hurried reader might take it as good news. The word that reframes it must therefore arrive early. The platform's caveat for that case opens with what would happen if you stopped marketing, then asks whether you believe it.

Remedies should be equally concrete. For a sampler that did not settle, the first advice is to run it again, since an unlucky start is sometimes the cause. If the failure repeats, the likely explanations are too few weeks or channels whose budgets move together, and both point toward data you can add.

Whether refusing makes a measurement tool less useful

Refusing does not make a measurement tool less useful, although it can feel that way on the day a fit is withheld. A tool that never refuses gives every fit the same look, so you cannot tell strong fits from weak ones. A tool that refuses lets each published fit stand for something.

Bayesian methodology already anticipates this behavior. The Bayesian workflow literature that the guide cites treats checking a model, and sometimes discarding it, as part of the method.

Refusal also preserves the honest interpretation. When data is too short or too flat, the truthful statement is that it cannot yet support an answer. Meridian's debugging guidance makes the same point from the other direction. With little information in the data, the posterior resembles the prior, so a tight-looking result may just echo assumptions.

Short answers on refused fits

Is a refused fit the same as a failed run?

No. A failed run crashes or times out, whereas a refused fit finished sampling and then failed a quality check. The refusal is a decision about the result. It comes with a written reason and a suggested remedy, such as adding weeks or varying spend across channels.

Will more data always fix a refused fit?

Not always, but more weeks and more variation in spend are the usual remedies. If channels still move together, or a channel stays flat, the data may not separate them. A holdout test or a deliberate budget change can then supply the missing contrast.

Why not publish the fit with a warning instead?

For most failures the model did not measure the thing asked, so no figure is worth caveating. A caveat fits when the model measured something and the answer is unusual. The split follows meaning rather than severity, which I think keeps warnings from turning into noise.

What should I do while I wait for more data?

Keep using the numbers you can defend, such as observed revenue and spend, and plan the data you need. Then work out what would change the answer, whether more weeks, more spend variation or an experiment, using the checklist in the guide.

Sources

• Stan Development Team, "How to Diagnose and Resolve Convergence Problems," mc-stan.org
• Google for Developers, "Model health checks," Meridian documentation
• Google for Developers, "Model debugging," Meridian documentation
• Measured, "How to QA an MMM: Top 5 Sanity Checks Before You Trust the ROI Numbers"
• Chan, D. and Perry, M., "Challenges and Opportunities in Media Mix Modeling," Google Research