01Identify what each calculation is comparing

Bland and Altman’s methodological article demonstrates why separate within-group significance tests can mislead readers of randomised trials. The intervention question requires comparing groups directly; comparing their p-value labels does not perform that test.Bland and Altman — Comparisons within randomised groups can be very misleading (opens in a new tab)

A within-group change compares a group with itself at an earlier time. A between-group comparison contrasts outcomes or changes between conditions. Both calculations can describe data, but they answer different questions.

The appropriate analysis depends on the design, including randomisation, pairing and baseline information. The table identifies questions rather than prescribing one universal analysis method.

02See why improvement in one group is incomplete evidence

In an original hypothetical experiment, group A changes from a mean of 40 to 50 signal units. Group B changes from 42 to 51. The observed changes are +10 and +9 units respectively.

The difference between changes is therefore +1 unit. Reporting only A’s ten-unit increase would omit the substantial change also seen in the comparator. The one-unit contrast is the descriptive difference between those changes, not a significance result.

Suppose instead that both groups change by ten units. Each group might show clear evidence of change over time, while the between-group change contrast is zero. A common time-related influence can affect both groups.

03Do not compare significance labels as if they were effects

Imagine that the +10 change in A has a p-value below the chosen threshold, while the +9 change in B does not. That difference in labels can arise from different precision, even when the estimated changes are close.

To assess whether the changes differ, the analysis needs the between-group contrast and its uncertainty. One small p-value and one larger p-value are not enough to reconstruct it.

The same issue can arise when a paper claims an effect only in one subgroup because that subgroup’s test was significant. Evidence that two effects differ requires a comparison of the effects, not merely separate tests of each against zero.

For a reader, the practical check is to find the exact test named in the caption or methods. Does it compare each condition with baseline, compare conditions with one another, or test a group-by-time relationship?

04Read baseline adjustment and design together

A paper may analyse follow-up values with baseline adjustment rather than subtracting raw group changes. That can be a valid way to address the intervention comparison, depending on the model and design. Do not assume a change-score calculation is always required.

Similarly, a raw difference between changes does not automatically establish causality in a non-randomised study. The groups may differ in other time-varying influences or in the process that determined their allocation.

A clear summary distinguishes the observed trajectories from the estimated intervention contrast. It gives the latter’s uncertainty and identifies whether the analysis accounted for relevant baseline and repeated-measure structure.

This preserves a useful observation that a group changed while avoiding a stronger claim that the intervention was responsible. The comparator and design provide the additional evidence needed to interpret that responsibility.