Biostatistics: Subgroup Analysis in Clinical Trials

Implications: The authors of this NEJM Special Report define subgroup analysis as “any evaluation of treatment effects for a specific end point in subgroups of patients defined by baseline characteristics.” When done appropriately, subgroup analysis are valuable. However, when done post hoc they can lead to incorrect conclusions. Understanding how subgroup analysis should be done can help when making decisions as to whether a company has correctly analyzed their clinical trials data or whether the results they are reporting have a high chance of being wrong.

Analysis:

In this paper a review of 97 clinical trials reported in the NEJM from July 2005-June 2006 showed 59 (61%) reported subgroup analysis. Of these it was unclear if the subgroup analysis were prespecified or post hoc (two thirds), whether interaction tests were used to assess heterogeneity (one half), and subgroup analysis results were not presented in a consistent way (one third). I will examine how each of these can impact how the results of a clinical trial are interpreted.

Documenting prespecified subgroup analysis in the protocol before patient recruitment involves clearly stating what outcome, patient subgroups, and statistical tests will be done. This allows the study sample size calculations to be made in order to achieve good power for the primary hypothesis and all the subgroup analysis. In addition, controlling the Type I error through an appropriate adjustment (e.g., Bonferroni) can be done to take into account the multiple testing. Post hoc analysis suffer from several drawbacks. First, they will often appear as ‘fishing expeditions’ and spun to appear as if the treatment effect seen in the subgroup analysis is real. Second, if enough subgroup analysis are performed (and articles rarely report how many were actually done) then by chance we expect to see some (i.e., 5%) of them showing a significant treatment effect that would not be replicated in an independent study.

Heterogeneity across subgroups means patients respond differently to a treatment and is appropriately tested by interaction terms in statistical models. The authors report how a test of the efficacy of pravastatin and baseline LDL was significant in reducing coronary events – pravastatin works better in patients with some levels of baseline LDL than in patients with other levels. It is statistically incorrect to declare heterogeneity based on differences in statistical tests in subgroups (e.g., significant effect in men but not in women). Another common mistake is to declare heterogeneity across subgroups based on observed treatment effect sizes if the confidence limits are not reported.

Guidelines for consistent reporting are included in this paper and should be considered as ‘good statistical practice’ when evaluating clinical trial reports. These include indicating the total number of prespecified and post hoc subgroup analysis performed (not just those reported), the impact of these on Type I error, whether interaction effects were used in assessing heterogeneity, and present the results in a consistent format (e.g., forest plots).

 

Perhaps the most important guideline recommendation is not to over interpret subgroup differences – in these subgroup differences is the greatest risk of finding false positives.