Study Design
In this section, key elements of systematic reviews and meta-analysis, controlled trials, observational cohort, case-control and cross sectional studies, and case series and case reports, are described.
The most common types of study design are meta-analysis or systematic review, controlled trials, which are ideally randomised controlled trials, the observational study group which includes cohort, case-control and cross sectional studies, case series and case reports, and laboratory or experimental trials. Research projects to develop or compare diagnostic tests usually follow a cohort-type design, although randomised controlled trials are best to see if a diagnostic test improves actual patient outcome. There are a huge number of study designs, most of which are a sub-type of one of these classifications.
This post will give you an overview of study design and advantages and disadvantages of each.
Why bother to understand study design and be able to distinguish between different types? As a clinician, you will use this understanding to:
• critically evaluate reports from the published literature or elsewhere, for example from the media;
• select a suitable study design for your own research project.
The ‘evidence pyramid’ shows the theoretical strength of evidence reflected by various types of study.
It is important to remember that these classifications are a guide, not hard and fast rules. Strength of evidence is governed by the quality of the study, including internal and external validity, as well as the design. A well designed and conducted cohort study probably provides stronger evidence for a given proposition than a poorly conducted randomised controlled trial.
Terminology: exposures and outcomes
First of all, some background. Epidemiologists and clinical researchers talk about ‘exposures’ and ‘outcomes’. An exposure is the event or risk factor that happens FIRST and causes or influences what happens – the outcome.
An ‘exposure’ can be a medical or surgical treatment, such as a drug or type of surgical procedure. It can be a risk factor, which could be a literal exposure such as exposure to lead or sunlight, or a protective factor such as eating a healthy diet or sunscreen. Factors such as sex or age are exposures. One is ‘exposed’ to being male, or ‘exposed’ to being young. Exposures are also called ‘interventions’, you can understand this in the context of drugs, surgical procedures, diet, physiotherapy and a host of other things which might be recommended to patients.
The ‘outcome’ always comes after the exposure. For example chronic exposure to lead, particularly in children, causes neurological damage, here neurological damage would be the outcome. Eating a healthy diet would protect a person from obesity and cardiovascular disease.
Having fair skin is an example of a risk factor that would make a person susceptible to sunburn.
A basic understanding of the classic epidemiological randomised controlled trial and observational study designs are essential to understanding systematic reviews and meta-analysis.
Different types of study depend on how the study participants or subjects are selected, according to the exposure and outcome, and who allocates the study subject to which group. Note that epidemiological study designs are loosely distinguished from those traditionally derived from agricultural studies such as factorial and latin square designs, although there is considerable crossover.
Intervention studies and observational studies
Interventions studies are those in which the researcher allocates some type of intervention or treatment to two or more groups of subjects. The randomised controlled trial is the classic intervention study.
Observational studies are those in which the researcher merely observes the effect of some exposure on two or more groups of subjects. Cohort, case-control and cross sectional studies are the classic observational study designs, often known as ‘epidemiological’ studies.
Randomised controlled trials
In a controlled trial (sometimes known as a clinical trial, although ‘clinical trial’ is not a very precise term), the researcher allocates the study subjects into treatment and control groups.
In order to make a valid comparison of the outcome depending on which treatment a subject is assigned to, the study groups need to be similar at baseline. If the study is not randomised, there is a high probability that the groups will NOT be similar at baseline, i.e. the beginning of the study. Methods of allocating subjects to groups on the basis of, for example, every second patient, or patients in every second week, are not reliable in producing groups which are similar at baseline.
Randomisation
To overcome the problem of creating groups which are similar at baseline, subjects should be randomised to one group or another (usually a treatment and control group). The key point is that each subject must have an equal chance of being allocated to one group or the other. Be suspicious if you read a paper which does not state whether, or how, randomisation was done!
Example of a randomised controlled trial: MRC Trial of streptomycin for tuberculosis 1948
You can find many examples of good randomised controlled trials in your own field of interest. To give you an example of one of the early landmark randomised controlled trials, in 1948 the British Medical Research Council conducted a trial of streptomycin in tuberculosis. The famous epidemiologist and statistician Austin Bradford-Hill was on the comittee and performed the randomisation, where 55 patients were randomly allocated by the researchers to streptomycin plus bed-rest and 52 to bed-rest alone. Patients in the treatment group were given two grams of streptomycin daily intramuscular in four divided doses for four months. All patients received 6 months bed-rest. At the end of the trial only 7% of the treatment group had died, compared with 27% of the bed rest group. This difference in the proportion of deaths was statistically significant at p<0.01.
Key points:
- 107 patients, 55 allocated to streptomycin plus 6 months bed-rest and 52 to 6 months bed-rest alone
- 2g streptomycin daily divided: I/M four times daily for 4 months
- 4/55 (7%) of the streptomcin group vs 14/52 (27%) bed-rest alone group died before the end of 6 months P<0.01
These days we use computerised randomisation. The figure below shows you what a such a randomisation looks like. In this example, each subject in the study (identified by patient number) is randomised to either treatment A or B.
Features of randomised controlled trials
- Subjects are allocated to groups by the researcher
- Valid randomisation technique so that groups are similar at baseline (except for the intervention of interest)
- Subjects should not know which treatment they are getting (if they know they are getting a placebo they will expect it ‘not to work’) (Blinding)
- Researchers measuring the outcome should not know which treatment the study subject has had (If they know the subject has had the active treatment they might nudge the outcome up to a ‘better’ outcome unconsciously or even consciously (‘Double Blinding’).
Advantages of randomised controlled trials
- The ‘gold standard’ for evaluating therapies
- Avoid confounding – ‘confounders’ which might have caused the effect instead of the treatment are equally distributed in each group due to randomisation.
Disadvantages of randomised controlled trials
- Takes a lot of time and organisation (randomisation, blinding, staff)
- Expensive
- Unethical where the exposure might cause harm.
Subtypes of randomised controlled trials: parallel and crossover designs
There are many subtypes of randomised controlled trials. The main ones are:
- Parallel group – two or more groups run in parallel at the same time – this is the most common design.
- Crossover design – In a crossover study, each subject receives both the treatment and control intervention, with each subject acting as their own control. The order or treatment is randomised. Half the subject gets the active intervention first, and the other half get the placebo. After a ‘washout period’, subjects received whichever treatment they did not get during the first ‘arm’. This design is suited to conditions in which the severity of disease does not fluctuate over time, where the treatment is expected to have an effect in a relatively short time and to ‘wear off’ quickly, i.e. have a short duration of action. It is very important that there is an adequate washout period so that the treatment during the first period does not affect the outcome during the second period. In crossover studies, each subject acts as their own control, reducing variation, so sometimes crossover studies can be conducted with fewer subjects than parallel group trials. However analysis of crossover designs can be very complex.
There are many more complex designs based on the idea of random allocation.
Cohort studies
Features of cohort studies
- The researcher looks for subjects who receive the exposure but DOES NOT decide who will get the exposure
- The researcher then tries to find similar subjects who have not received the exposure
- SUBJECTS ARE SELECTED ON EXPOSURE
- and a NON-EXPOSED group selected
- The outcome is determined after following the subjects over time.
Cohort studies are similar to randomised controlled trials in concept, except that the researcher does not allocate the exposure.
Many of you have seen this figure before, it depicts two groups. The line indicates a timeline at time zero, the start of the study, through to time 2 and beyond. The two groups are followed over time and then the outcome determined.
Example of a cohort study: the British Doctors Study
The British Doctors Study, conducted by the British Medical Research Council during the 1950’s, is the classic cohort study. At the time there was controversy about whether or not smoking was harmful. It would have been unethical to randomise subjects to either smoke or not smoke, so the researchers selected a well-defined group, British doctors, and divided them into a ‘smoking’ and ‘non-smoking’ cohort, and followed them over time. By 1954 it was apparent that smoking was related to a number of adverse outcomes, including premature death, lung cancer, myocardial infarction and respiratory disease.
The reports have been published in 1954, 1956 and 2004.
Advantages of cohort studies
- Can calculate relative risk and disease odds ratio
- One advantage of a cohort study is that one can infer causation, because the hyothesised ‘cause’ comes temporally before the outcome, i.e. comes before the outcome in time, similar to a randomised controlled trial.
- Incidence of disease in exposed and unexposed subjects can be calculated
- In prospective studies, data collection is well designed and complete and bias due to faulty recall of events, particularly exposures, is minimised.
- It is theoretically possible to undertake a retrospective cohort study, if there are very good records and complete databases available.
Disadvantages of cohort studies
- Exposed and unexposed proportions in the target population cannot be estimated
- Unsuitable for rare diseases because large numbers of subjects would need to be studied
- Potentially long duration of follow-up
- Dropouts to follow-up are likely & maintaining follow-up is difficult
- Control of extraneous variables may be incomplete
- Potentially expensive
Case-control studies
Features of case-control studies
- SUBJECTS ARE SELECTED ON DISEASE
- and a comparable NON-DISEASED group selected
- Cases selected first
- Controls selected as close as possible to cases except for disease or outcome of interest
- (i.e. controls are ‘matched’ to the cases either as a group or on an individual basis)
- ‘Look backward’ to see what the exposure was
Example of a case-control study: Helicobacter pylori & gastritis
Barry Marshall, a young resident at the Royal Perth Hospital, and Robyn Warren, a pathologist at the same hospital, were interested in a spiral bacteria they had noticed on endoscopic gastric biopsies of people with gastritis. They hypothesised that these hitherto unknown and unidentified bacteria were associated with gastritis. From a sample of 100 people undergoing gastroscopy for various reasons, they selected those with gastritis and those without gastritis, and compared their exposure to the as yet unclassified organism.
They found a significant association between gastritis and the presence of bacteria in the biopsy specimen (p<0.0001).
This data was published in the Lancet in 1984. The pair, having had their initial abstract rejected at an a local scientific conference in 1983, went on to win a Nobel Prize. See (https://www.nhmrc.gov.au/media/podcasts/2009/conversation-professor-barry-marshall).
Advantages of case-control studies
- Good for rare diseases
- Good for diseases with long incubation periods
- Quick to organise and conduct
- Relatively cheap
- Require relatively few subjects
- Sometimes can use existing records
- No risk to subjects
- Allow assessment of multiple exposures
Disadvantages of case-control studies
- Can’t estimate exposed and unexposed proportions in target population
- May be recall bias for exposure
- May be hard to validate exposure
- Incomplete control of extraneous variables
- Difficult to select control group
- Incidence of disease in exposed and unexposed subjects can’t be estimated
Cross sectional studies (surveys)
Features of cross sectional studies
- In a cross sectional study, A REPRESENTATIVE SAMPLE OF THE POPULATION IS TAKEN
- Exposure and outcome are ascertained after data collection
- May be cross sectional – undertaken over a short specified time period – a single ‘snapshot’ in time (eg prevalence surveys)
- May be longitudinal – observations are repeated over time to provide information about the course of disease over time and space (eg incidence risk or incidence rate)
Example of cross sectional study: oral health in rural children
Ha and colleagues wished to evaluate a number of risk and causative factors related to a number of oral health outcomes in Australian children. They conducted a large survey and then analysed the data for associations between factors such as rurality, indigenous status, being from a fluoridated water area, and socioeconomic status with outcomes such as number of teeth with caries, and decayed, missing or filled deciduous teeth.18
Advantages of cross sectional studies
- If a random sample of the target population is selected, can estimate prevalence and proportion of exposed and unexposed subjects in the target population
- Quick to conduct
- Relatively cheap
- Sometimes can use current records
- No risk to subjects
- Can assess multiple exposures and outcomes
Disadvantages of cross sectional studies
- Unsuitable for rare diseases
- Unsuitable to diseases of short duration
- Hard to control extraneous variables
- Can’t estimate incidence in exposed and unexposed individuals
- Can’t determine cause and effect (temporal exposure to outcome sequence can’t be evaluated)
Case series and case reports
There are just a few comments on case series and case reports. These type of reports have no control group, so effects of treatment are very unreliable because disease may fluctuate or resolve naturally.
Case series and case reports are very important in flagging the emergence of new diseases and recognition of syndromes. There are many examples of important conditions which were identified initially through case series or case reports, including
- HIV/AIDS
- Hendra Virus disease
- Cystic Fibrosis
Laboratory or experimental studies
In the context of human medicine, laboratory studies or experimental studies are usually like a randomised controlled trial but conducted under artificial, controlled circumstances. There studies are generally conducted in species other than humans, and even for those species, are not ‘real world’ as the animals are husbanded under very artificial conditions and indeed are usually themselves very genetically homogeneous and often genetically modified.
These studies are very important for basic science but are not generalisable directly to clinical practice.
Systematic reviews
In a systematic review, all relevant studies have been systematically identified, appraised and summarised using explicit and reproducible methods (search terms, databases).
Meta-analysis
In a meta-analysis, results from all relevant studies are systematically identified, appraised and summarised using explicit and reproducible methods as for a systematic review. Additionally, the results for each included report are analysed statistically to give an overall summary result.
Example of meta-analysis: streptokinase for acute myocardial infarction.
Joseph Lau and colleagues in 1992 conducted a meta-analysis of mortality after treatment of myocardial infarction with intravenous streptokinase. Such studies had been carried out from 1959 and were still being carried out at the time of publication of the analysis in 1992. The first block represents a conventional meta-analysis, representing nearly 37,000 patients, arranged in order of year of publication from the top. The odds ratio for each individual study is represented as a horizontal line. Lines which do not cross one (the vertical line) are ‘statistically significant’ in their own right. The overall pooled estimate at the lower left shows that streptokinase is highly effective at reducing mortality.
The authors then performed a cumulative meta-analysis, using special techniques, which showed that by the time 2,432 patients had been evaluated, there was a clear advantage of streptokinase; this would have been known by 1973, years before ‘clotbusters’ became routine therapy for myocardial infarction.
One lesson from this is that in many cases, it may be important to do a meta-analysis before conducting a new study – the answer may already be out there!