Methodology and FAQ

This page explains how the Sproxil Nigeria Malaria Survey moves from raw responses to published indicators, why the survey is calibrated, what the wealth scale shows, how robust the estimates are, and what a realistic standard of success is for a survey of this design. Every external claim is backed by a peer-reviewed reference linked in the References section.

Executive summary

SNMS is a large phone survey that reaches a more connected, health-active, and wealthier slice of Nigeria than a national household survey does, so its raw numbers describe that engaged population, not the whole country. To compare it fairly with the national surveys, each indicator is rebuilt on the comparator's own definition and the sample is calibrated to that comparator's population. The decisions behind this, each evidenced in the sections below, are:

• We benchmark against MIS 2021 (primary) and NDHS 2023-24 (secondary), and align every indicator to the comparator's own definition, because the two surveys define several indicators differently. See methodology selection.
• We report three values per indicator: SNMS unweighted, SNMS raked to MIS shown against MIS 2021, and SNMS raked to DHS shown against NDHS 2023-24. See the decision pipeline.
• Because SNMS is a non-probability sample, we calibrate by raking to each comparator's population margins rather than using design weights. See the limitation that matters.
• Raking to the MIS wealth scale is statistically expensive but does not move the estimates, so we trim the MIS weights at a 20x cap, a choice supported by a full sensitivity analysis. See the raking sensitivity analysis.
• No wealth reweighting removes the engaged-population selection, and the post-2021 macroeconomic upheaval affects health-seeking behaviour in ways no wealth scale captures. See the wealth scale and the macroeconomic question.
• The realistic and appropriate success standard for this design is substitute with adjustment, not direct substitution. See what success looks like.

1. How to read SNMS, in one paragraph

SNMS is a large, continuously fielded phone survey. It reaches a more connected, more health-active, wealthier slice of Nigeria than a national household survey does. That is its nature, not a flaw, but it means the raw headline numbers describe the engaged population the survey reaches, not the whole country. To compare SNMS with the national surveys, each indicator is rebuilt on the same definition the national survey uses and then calibrated to national distributions. After that correction, SNMS provides a usable substitute for most core indicators, a complementary signal for a few, and three measures no national survey carries at all.

2. Methodology selection

This edition benchmarks SNMS against the Nigeria Malaria Indicator Survey 2021 as the primary reference, with the Nigeria Demographic and Health Survey 2023-24 as a secondary reference. The MIS is the malaria-specific national survey and its indicator definitions are the closest match to the SNMS questionnaire. The NDHS is more recent and is shown alongside so readers can see movement since 2021.

Two design facts shape everything else. First, SNMS is a non-probability sample: respondents are reached by phone, not by a random draw from every household, so design-based estimation does not apply and the sample must be adjusted by modelling instead (Mercer et al. 2017). Second, the adjustment is calibration to national distributions of wealth, geographic zone, and urban or rural residence. Calibration reduces the distortion from the sample being skewed, but it cannot remove all of it, for reasons set out in section 5.

3. From raw responses to published indicators

The pipeline is the same for every indicator.

• Raw responses. Each respondent answers a fixed questionnaire with skip logic, so only the relevant subgroup sees each module (for example, only women who report a recent birth answer the antenatal-care questions).
• Definition alignment. For each indicator, the numerator and denominator are built to match the national-survey definition. For example, antenatal care from a skilled provider counts only a doctor or a nurse or midwife, and the case-management indicators are computed only for children who had a fever in the last two weeks.
• Unweighted headline. The indicator is first computed as collected. This is the engaged-population reading.
• Calibration. The sample is reweighted to national distributions of wealth, zone, and residence, and the indicator is recomputed. This is the figure used for comparison with the national surveys.
• Verdict. The indicator is rated under the Cross-Survey Indicator Alignment Framework (CSIAF) as a substitute with adjustment, a complementary signal, or a new measure.

4. The wealth scale and the macroeconomic question

A natural worry is that Nigeria's economy changed so much after 2021 that an old wealth scale is misleading. The data give a careful answer.

The raw SNMS sample is heavily skewed toward the wealthiest. On the DHS-anchored wealth scale, two thirds of respondents fall in the richest fifth and under one percent in the poorest fifth. Calibration corrects this. The chart on this page shows three distributions across the five wealth quintiles:

The raw sample is heavily skewed toward the wealthiest quintile. Calibrating to national distributions corrects this. The MIS 2021 and NDHS 2023-24 wealth targets are nearly identical, so the choice between them barely changes the wealth adjustment. Wealth quintiles are relative measures and are robust to macroeconomic shifts; the limitation that matters is that no wealth reweighting removes the engaged-population selection, and the cost-of-living changes since 2021 affect health-seeking behaviour in ways no wealth scale captures.

Two things follow. First, the MIS 2021 and DHS 2023-24 wealth targets are nearly identical, so the choice between them barely changes the wealth adjustment. This is expected: a wealth index is a relative measure that ranks households within a survey rather than an absolute measure of income, so the quintile shares are robust to economy-wide price changes (Filmer and Pritchett 2001; Rutstein and Johnson 2004).

Second, the scale itself did move, modestly. In plain terms, the same household looks slightly less rich on the newer scale, which is the footprint of changing asset ownership and prices since 2021. The effect is real but small, and because the calibration targets barely differ, it does not change the corrected indicator values much.

The honest conclusion is therefore not that the 2021 wealth weights are obsolete. It is that wealth reweighting, whether anchored to 2021 or 2023-24, is not where the limitation lies.

5. The limitation that matters

No wealth reweighting removes the engaged-population selection. A phone survey reaches people who differ from non-respondents in ways a wealth scale does not capture, above all in health-seeking behaviour and access. The strongest evidence comes from a 34-country study of phone surveys against the Demographic and Health Surveys: poststratifying on age, education, residence, wealth, and marital status corrected under-5 mortality estimates, but fertility estimates stayed biased, because the behaviours that drive the gap, such as contraceptive use, are not captured by those weighting variables (Sanchez-Paez et al. 2023). The same logic applies to SNMS: calibrating on wealth, zone, and residence corrects the part of the gap those variables explain, and leaves the part driven by who chooses to answer a malaria phone survey. This is why a residual gap to the national surveys remains on the engagement-sensitive indicators even after calibration, and why SNMS values are presented as an engaged-population reading with a correction, not as a national headcount.

This is a general result, not a quirk of SNMS. Reviews of non-probability surveys find that weighting reduces but does not reliably remove selection bias, and that valid correction depends on having measured every variable that drives both selection and the outcome, an assumption that rarely holds in full (Mercer et al. 2017; Cornesse et al. 2020).

6. Robustness and uncertainty

Calibration is not free. Reweighting a skewed sample inflates variance, because a small number of under-represented respondents carry large weights. The standard measure is the design effect. Using Kish's formula, the design effect equals one plus the squared coefficient of variation of the weights (Kish 1992; Valliant, Dever and Kreuter 2018).

7. The raking sensitivity analysis: why the MIS weights are trimmed at 20x

SNMS is a non-probability sample, so it is calibrated to each comparator's population by raking. Raking to the MIS 2021 margins on the MIS wealth scale produces a very large design effect of 45.5 (effective sample about 1,760), because the MIS scale places only 57 of the 80,269 respondents in the poorest fifth, and those few must then be inflated to weights near 392 times the mean to reach the 16 percent national target.

To choose a defensible weighting rather than asserting one, we trimmed the weights at a series of caps (each a multiple of the mean weight), re-raked after each cap, and measured both the design effect and how far the weighted wealth distribution drifts from the MIS target.

Two findings follow. First, the design effect falls steeply as the weights are trimmed, from 45.5 at full raking to 6.6 at a 20x cap, while the effective sample rises from about 1,760 to over 12,000. Second, and decisively, the indicator values barely move across the entire range, by one to two points at most, even as the poorest-fifth match collapses from 16.2 percent to under 2 percent. The eight-to-eleven-point gap that matters, between the unweighted values and any raked version, is already present at the first cap and is produced by the zone and urban-rural adjustments, not by wealth. The margin drift introduced by trimming is confined to the wealth bottom quintile; zone and urban-rural stay matched throughout.

The conclusion is that the wealth raking is statistically expensive but substantively inert: it is the entire source of the design effect, yet it changes the estimates by a fraction of a point. We therefore trim the MIS weights at a 20x cap, which brings the MIS-raked design effect to 6.6, essentially identical to the 6.8 of the DHS raking, lifts the effective sample to about 12,000, and changes the MIS-raked values by about a point relative to the fully matched version. The only quantity given up is an exact wealth match that was not affecting the estimates, so the trade is a large gain in precision at no material cost to accuracy (Kish 1992; Valliant, Dever and Kreuter 2018).

8. What success looks like

Because SNMS and the national surveys use fundamentally different designs, a phone-reached non-probability sample on one side and an in-person probability sample with household listing on the other, the two were never going to produce identical values that allow one to drop in for the other unchanged. Direct substitution was not an achievable or expected outcome, and the platform does not claim it. The realistic and appropriate standard of success for a survey of this design is substitute with adjustment: SNMS measures the same construct as the national survey, and once the definition is matched and the sample is calibrated, the corrected SNMS value is usable for tracking and comparison. On the current assessment, 8 of the 11 comparable indicators meet that standard, 3 are complementary signals, and 3 are measures no national survey carries. That is the success criterion this survey should be judged against, and it is consistent with the survey-methods literature on what non-probability samples can and cannot deliver (Mercer et al. 2017; Cornesse et al. 2020; Sanchez-Paez et al. 2023).

9. How SNMS can strengthen its statistical footing

The same literature points to concrete, fundable improvements.

• Measure the variables that drive selection. The clearest lesson of the 34-country study is to collect the behavioural and access characteristics that differ between respondents and non-respondents, so they can enter the calibration and shrink the residual gap (Sanchez-Paez et al. 2023).
• Add a birth-recency question so antenatal and IPTp indicators can use the national two-year window rather than the current five-year window.
• Add a per-child net-use question so under-5 ITN use becomes measurable.
• Anchor calibration to a small probability sub-sample. A modest, in-person or address-based probability sample fielded periodically gives a benchmark to calibrate the phone sample against, the dual-frame approach recommended for non-probability data (Cornesse et al. 2020).
• Oversample under-represented groups. Targeted recruitment of poorer and rural respondents reduces the extreme weights that drive the design effect, raising the effective sample size.
• Report uncertainty honestly. Publish confidence intervals, effective sample sizes, and the design effect alongside every estimate, and suppress cells that rest on too few respondents.

Frequently asked questions

References

1. Sanchez-Paez DA, Masquelier B, Menashe-Oren A, Baruwa OJ, Reniers G. Measuring under-5 mortality and fertility through mobile phone surveys: an assessment of selection bias in 34 low-income and middle-income countries. BMJ Open. 2023;13(11):e071791. https://doi.org/10.1136/bmjopen-2023-071791
2. L'Engle K, Sefa E, Adimazoya EA, Yartey E, Lenzi R, Tarpo C, et al. Survey research with a random digit dial national mobile phone sample in Ghana: methods and sample quality. PLoS ONE. 2018;13(1):e0190902. https://doi.org/10.1371/journal.pone.0190902
3. Mercer AW, Kreuter F, Keeter S, Stuart EA. Theory and practice in nonprobability surveys: parallels between causal inference and survey inference. Public Opinion Quarterly. 2017;81(S1):250-271. https://doi.org/10.1093/poq/nfw060
4. Cornesse C, Blom AG, Dutwin D, Krosnick JA, De Leeuw ED, Legleye S, et al. A review of conceptual approaches and empirical evidence on probability and nonprobability sample survey research. Journal of Survey Statistics and Methodology. 2020;8(1):4-36. https://academic.oup.com/jssam/article/8/1/4/5699631
5. Kish L. Weighting for unequal Pi. Journal of Official Statistics. 1992;8(2):183-200. See also Valliant R, Dever JA, Kreuter F. Practical Tools for Designing and Weighting Survey Samples. 2nd ed. Springer; 2018. https://link.springer.com/book/10.1007/978-3-319-93632-1
6. Filmer D, Pritchett LH. Estimating wealth effects without expenditure data or tears: an application to educational enrollments in states of India. Demography. 2001;38(1):115-132. https://doi.org/10.1353/dem.2001.0003
7. Rutstein SO, Johnson K. The DHS Wealth Index. DHS Comparative Reports No. 6. Calverton, MD: ORC Macro; 2004. https://dhsprogram.com/pubs/pdf/cr6/cr6.pdf

Question routing and denominator construction

The SNMS questionnaire applies conditional question routing. Respondents are presented only with questions that apply to their household composition and prior answers. For example, questions about under-five net use are asked only of respondents who report a child under five in their household; questions about IPTp uptake are asked only of respondents reporting a pregnancy in the relevant recall window; questions about R21 vaccine receipt are asked only of respondents whose age-eligible children fall within the cohort covered by the routine immunisation programme.

This routing means each indicator has its own analytic universe, and that universe rarely equals the full set of 80,269 respondents. All SNMS values displayed throughout this site are computed against the correct routing-aware denominator for each indicator, so the SNMS estimates can be compared like-for-like with the MIS 2021 and NDHS 2023-24 reference values.

Respondent anonymity and research ethics

• Informed consent is obtained from each SNMS respondent at the start of every CATI and CAWI interview. The consent script is available on request.
• No individual respondent is identifiable on this Site.
• All values displayed are aggregated to a minimum cell size of 25 respondents. Values for cells below this threshold are suppressed and replaced with a small-cell notation.
• The lowest geographic resolution displayed is the LGA. No sub-LGA values (ward, locality, enumeration area) are shown.
• Re-identification of any respondent is prohibited under the Terms of Use.

Contacts for researchers

• General SNMS inquiries: snms@sproxil.com
• Research methodology, indicator definitions, future microdata access: snms-research@sproxil.com
• Privacy and data protection: privacy@sproxil.com
• Media inquiries: communications@sproxil.com

How to cite

See also the CSIAF Methodology document.