01Identify the quantity being compared

Cochrane distinguishes difference measures from ratio measures and describes mean differences and standardised mean differences for continuous outcomes. These measures express comparisons in different ways; they should not be read as interchangeable numerical scores.Cochrane Handbook — Choosing effect measures and computing estimates of effect (opens in a new tab)

A mean difference retains the outcome’s units. If the measured outcome is a defined assay signal, the difference is expressed in that signal’s units. Its biological interpretation still depends on what the assay measures and how directly it reflects the question of interest.

Direction needs interpretation too. A positive difference could represent more activity, more damage or simply a higher instrument signal. A plus sign is not inherently a favourable biological result.

02Separate raw magnitude from standardised magnitude

In an original hypothetical comparison, the mean signal is 101 units in the reference group and 109 units in the peptide condition. Subtracting reference from peptide gives an estimated difference of +8 units.

Suppose two illustrative datasets have that same difference but different pooled standard deviations. With a pooled SD of 4 units, the difference divided by that SD is 2. With a pooled SD of 16 units, it is 0.5. The raw difference remains 8 units in both cases.

This calculation shows why a standardised effect changes with variability as well as with the difference itself. It does not demonstrate that the first biological change is four times as important as the second.

03Ask what magnitude would matter for this question

NC3Rs advises distinguishing an effect worth detecting on biological grounds from an effect merely observed in earlier data. Its sample-size guidance connects the relevant difference to the study’s primary outcome, rather than letting a statistical result supply the definition of importance afterwards.NC3Rs EDA — Group and sample size (opens in a new tab)

For a reader, the corresponding task is to find the rationale. Did the authors explain why a change of this size would alter their interpretation of the model, support a follow-up question or distinguish competing explanations?

A generic small, medium or large label does not answer that question. Eight signal units could be consequential in one measurement system and trivial in another. The instrument scale, baseline behaviour and relation to the underlying phenomenon all matter.

Nor does a large effect on a proxy automatically imply a large functional effect. An estimate is specific to the measured outcome until additional evidence establishes the connection to a broader biological claim.

04Read the estimate and its uncertainty together

A point estimate is the centre of a statistical summary, not an exact known property of the material. Two studies with an estimated difference of 8 units can support very different conclusions if one estimate is precise and the other leaves a wide range of effects unresolved.

Check whether the estimate is adjusted for other variables, whether it concerns all experimental units or a subgroup, and whether it was selected from several outcomes. These choices define what the number summarises.

A clear evidence note names the outcome, comparator, estimated effect and uncertainty, followed by the relevant setting. It then distinguishes the observed magnitude from the authors’ judgement about importance. This preserves information that disappears when a paper is reduced to significant or not significant.