Effect size

You must cite this article if you use its information in other circumstances. An example of citing this article is:
Ronny Gunnarsson. Effect size [in Science Network TV]. Available at: https://science-network.tv/effect-size/. Accessed November 22, 2019.
Suggested pre-reading What this web page adds
  1. Introduction to statistics
  2. Observations and variables
  3. Inferential statistics
  4. Choosing statistical analysis
This web-page provides an introduction to the concept of effect size. Reading this will give you an understanding of what effect size is and the different types.
Is your effect size high enough? (Photo by John Chapman)

This page will explain the difference between “statistical significance” and “effect size”, the latter also labelled clinical significance in health care. The label “effect size” is most often used when discussing effect of an intervention in study designs comparing groups. However, it is also used in observational studies trying to explore effects.

P-value versus effect size

We often talk about the p-value. P stands for “probability”. What kind of probability is it? In the situation of comparing two interventions the p-value states the probability of being wrong when we say that treatment A has a larger effect than treatment B. A low p-value says it is unlikely that we would get the observed observations if the difference in effect between groups is zero. You can read more about this on the page Level of significance. It is important to remember that the p-value only talks about statistical significance but says nothing about importance (clinical significance in health care).

In large studies we may reach statistical significance but the difference between groups may be so small that is is of no importance (no clinical significance). Hence, we need another measure than the p-value showing the importance (clinical significance) of the differences in effect between treatments. This measure is labelled “Effect size”. The p-value is always a probability estimated with a figure between 0-1. The effect size is different. It is not a single figure, rather many different estimates that may all be labelled as effect size when used to evaluate effect of interventions.

Different types of effect size

We use different types of effect sizes depending on the outcome variable and if the scenario is group comparisons with no need to adjust for effect modifiers or confounding variables or, if the scenario focus on associations, the latter commonly used also for group comparisons where there is a need to adjust for effect modifiers and confounding variables. You may find the web page about Choosing statistical analysis clarifying. The most common types of effect sizes are:

Type of effect sizeEffect size
A. Difference in outcome between groups* where the outcome is measured by a continuous scale (such as blood pressure)Cohen’s d, Glass’ Δ, Hedges’ g, The Drug-Placebo Response Curve , eta-squared**, partial eta-squared** Omega Squared.
B. Difference in outcome between groups* where the outcome is binary (yes/no or 0/1)Relative risk reduction (RRR), Absolute risk reduction (ARR), Number needed to treat (NNT), Number needed to harm (NNH), Non occurrence probability increase (NOPI) .
C. Correlation between an independent variable and a dependent variable measured by a continuous scaleR**R-squared**Beta-coefficient**
D. Correlation between an independent variable and a dependent variable that is binary (yes/no or 0/1)Odds ratio (OR)**, Relative risk (RR)**, Hazards ratio (HR)**

*Group allocation means a variable determining if participants had one intervention / exposure compared to another / or placebo.
**Use techniques where you can adjust for confounding variables.

The relation between p-values and effect size

Cohen’s d is sometimes referred to as the effect size and this may cause confusion. It is important to note that Cohen’s d is only one of several different estimates of effect size and Cohen’s d is not always suitable (see table above). Furthermore, it is common to have more than one outcome measure so you may have to mix different types of estimates of effect size.

P-value is given as a single estimate. However, the effect size is given as a point estimate with a confidence interval (usually a 95% confidence interval). The p-value and the effect size are tied to each other so when the p-value increases and approaches 0.05 the 95% confidence interval for effect size will approach the limit for no effect. Hence, the p-value and the effect size are merely the opposite sides of the same coin.

Magnitude of effect size

The table below are rough rules of thumb that should be adjusted for your particular context.

 No effectVery smallSmallMediumLargeVery largeHuge
Cohens d*<0.010.01-0.19 0.20-0.49  0.50-0.79 0.80-1.1 1.2-1.9 2.0- 
R* 0.0 0.10-0.29 0.30-0.49 0.50-   
R squared 0.0  0.010-0.0890.090-0.24 0.25-  
Odds ratio**1.0 1.5-3.5 or 0.29-0.673.5-4.9 or 0.20-0.295.0 –  or <0.20  
Hazards ratio**1.0      

* Irrespective if the effect size is positive or negative.
** Near 1.0 is no effect. The more it moves away from 1.0 (upwards or downwards) the larger the effect.

Cohens d

(This section is still under construction. Sorry for the inconvenience.)

Hedge’s g

(This section is still under construction. Sorry for the inconvenience.)

Useful links

References

1.
Chen H, Cohen P, Chen S. How Big is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies. Communications in Statistics - Simulation and Computation [Internet]. 2010 Mar 31 [cited 2017 Mar 27];39(4):860–4. Available from: http://dx.doi.org/10.1080/03610911003650383
1.
Cohen J. Statistical power analysis for the behavioral sciences. Second edition. Hillsdale: Routlegde; 1988.
1.
Sawilowsky S. New Effect Size Rules of Thumb. Journal of Modern Applied Statistical Methods [Internet]. 2009 Nov 1;8(2). Available from: http://digitalcommons.wayne.edu/jmasm/vol8/iss2/26
1.
Hagström B, Gunnarsson RK, Rosenfeld M. Presenting the improved possibility for staying well might be better than talking about change in risk: Use of the Non-Occurrence Probability Increase (NOPI). Scandinavian Journal of Primary Health Care [Internet]. 2013 Sep 1 [cited 2017 Mar 15];31(3):138–40. Available from: http://dx.doi.org/10.3109/02813432.2013.811951
1.
Faraone SV, Biederman J, Spencer TJ, Wilens TE. The drug-placebo response curve: a new method for assessing drug effects in clinical trials. J Clin Psychopharmacol. 2000 Dec;20(6):673–9.
You must cite this article if you use its information in other circumstances. An example of citing this article is:
Ronny Gunnarsson. Effect size [in Science Network TV]. Available at: https://science-network.tv/effect-size/. Accessed November 22, 2019.

Comments are closed.