This web-page will give you an understanding of what pilot studies (feasibility studies) are. It will also clarify the difference between a planned pilot study and an underpowered study.
Chief investigators in a pilot study? Photo by Alex Pereslavtsev
This page aim to clarify the difference between a pilot study and other studies. This discussion is most often relevant in relation to studies taking on an empirical-atomistic approach aiming to evaluate the effect of a drug or any other intervention. However, it can be used also for observational studies. The rest of this page will focus the discussion to the situation of pilot studies preparing the way for a larger study to establish effect of an intervention.
It is a common misconception that small underpowered studies trying to prove effect of an intervention compared to placebo or another intervention are pilot studies . This is not true. If they aim to prove effect and are underpowered it means that they either failed to recruit participants up to their pre-calculated target or it was simply a badly designed study, not a pilot study. A pilot study aim to answer questions such as:
Will a large or small proportion of individuals asked to participate accept participation? Will randomisation make potential participants to hesitate?
Will the measurement of effect be practical and feasible? Which survey is suitable in this situation? Should a new survey tool be constructed?
Is the intervention reasonably easy to apply given the resources we could expect to have in a larger study?
Will a large or small proportion of patients drop out of the study for various reasons?
What level of random variability and rough changes or effects can be seen after intervention? (to be used as a basis for sample size estimation before the larger adequately powered study)
How common are unforeseen problems (that may warrant exclusion of a patient or worse termination of the study)?
A pilot study aim to provide answers to some or all of these questions but is not powered to answer the big question is there an effect of the intervention. Hence, the reporting of a true pilot study should completely refrain from trying to reject a null hypothesis concerning possible effect of the studies intervention. Effect size may be calculated only to serve as a guide for a sample size calculation before the final study.
Pilot study
“The real thing”
Sample size calculation before data collection
Rarely (see below)
Should always be done
Number of included participants
Usually very small
Usually fairly large
Randomization
Often if it is an interventional study
Should be done if the focus is to evaluate effect of something
Aim to establish effect
Never ! (but may make a preliminary estimation of effect size to inform a sample size calculation for the final study.)
Is only done to inform a sample size calculation for the coming final study.
Should always be used to answer the research questions.
Sample size in pilot studies
Various recommendations point to a number between 10-50 or higher. However, a proper sample size calculation can only be done for research question VI above . There is no sensible way to make a proper sample size calculation for research question I-V.
Effectiveness and efficacy
Efficacy is a measure of effect of the intervention under ideal circumstances while effectiveness is a measure of the effect if implemented in reality. Pilot studies does not aim to estimate efficacy or effectiveness.
Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology [Internet]. 2010 Jan 6;10:1. Available from: https://doi.org/10.1186/1471-2288-10-1
