[R-meta] Sample size and continuity correction

Gerta, In the case of two studies there is a caveat w.r.t. to the 
overlapping CI heuristic (probably also in the three study case, but I 
do not know a number for that):

If, say, the assumptions of the two-sample t-test hold, then the CIs 
might overlap, but the t-test might be significant. The significance 
of the t-test might be seen as an indicator of heterogeneity. 
Goldstein and Healy (1995) argue in favour of 83% CIs because of this 
suggestion (I am not sure I buy into that) and there is also a note by 
Cumming and Finch (2005). Even if the assumptions of the two-sample 
t-test do not hold, but appropriate CIs are available, the "overlap 
but significant differences" might still hold.

Harvey Goldstein; Michael J. R. Healy. The Graphical Presentation of a 
Collection of Means, Journal of the Royal Statistical Society, Vol. 
158, No. 1. (1995), p. 175-177.

Cumming, Geoff; Finch, Sue. Inference by Eye: Confidence Intervals and 
How to Read Pictures of Data, American Psychologist, Vol 60(2), 
Feb-Mar 2005, p. 170-180.


-Philipp



On Thu, Aug 27, 2020 at 9:24 PM Gerta Ruecker 
<ruecker at imbi.uni-freiburg.de <mailto:ruecker at imbi.uni-freiburg.de>> 
wrote:

    Dear Nelly and all,

    With respect to (only) the first question (sample size):

    I think nothing is wrong, at least in principle, with a
    meta-analysis of
    two studies. We analyze single studies, so why not combining two of
    them? They may even include hundreds of patients.

    Of course, it is impossible to obtain a decent estimate of the
    between-study variance/heterogeneity from two or three studies.
    But if
    the confidence intervals are overlapping, I don't see any reason to
    mistrust the pooled effect estimate.

    Best,

    Gerta



    Am 27.08.2020 um 16:07 schrieb ne gic:

    > Many thanks for the insights Wolfgang.
    >
    > Apologies for my imprecise questions. By "agreed upon" & "what
    > conclusions/interpretations", I was thinking if there is a

    minimum sample

    > size whose pooled estimate can be considered somewhat reliable

    to produce

    > robust inferences e.g. inferences drawn from just 2 studies can be
    > drastically changed by the publication of a third study for

    instance - but

    > it seems like there isn't. But I guess readers have to then

    check this for

    > themselves to access how much weight they can place on the

    conclusions of

    > specific meta-analyses.
    >
    > Again, I appreciate it!
    >
    > Sincerely,
    > nelly
    >
    > On Thu, Aug 27, 2020 at 3:43 PM Viechtbauer, Wolfgang (SP) <
    > wolfgang.viechtbauer at maastrichtuniversity.nl

    <mailto:wolfgang.viechtbauer at maastrichtuniversity.nl>> wrote:

    >

    >> Dear nelly,
    >>
    >> See my responses below.
    >>

    >>> -----Original Message-----
    >>> From: R-sig-meta-analysis [mailto:

    >> r-sig-meta-analysis-bounces at r-project.org

    <mailto:r-sig-meta-analysis-bounces at r-project.org>]

    >>> On Behalf Of ne gic
    >>> Sent: Wednesday, 26 August, 2020 10:16
    >>> To: r-sig-meta-analysis at r-project.org

    <mailto:r-sig-meta-analysis at r-project.org>

    >>> Subject: [R-meta] Sample size and continuity correction
    >>>
    >>> Dear List,
    >>>
    >>> I have general meta-analysis questions that are not
    >>> platform/software related.
    >>>
    >>> *=======================*
    >>> *1. Issue of few included studies *
    >>> * =======================*
    >>> It seems common to see published meta-analyses with few

    studies e.g. :

    >>>
    >>> (A). An analysis of only 2 studies.
    >>> (B). In another, subgroup analyses ending up with only one

    study in one of

    >>> the subgroups.
    >>>
    >>> Nevertheless, they still end up providing a pooled estimate in

    their

    >>> respective forest plots.
    >>>
    >>> So my question is, is there an agreed upon (or rule of thumb,

    or in your

    >>> view) minimum number of studies below which meta-analysis becomes
    >>> unacceptable?

    >> Agreed upon? Not that I am aware of. Some may want at least 5

    studies (per

    >> group or overall), some 10, others may be fine with if one

    group only

    >> contains 1 or 2 studies.
    >>

    >>> What interpretations/conclusions can one really draw from such

    analyses?

    >> That's a vague question, so I can't really answer this in

    general. Of

    >> course, estimates will be imprecise when k is small (overall or

    within

    >> groups).
    >>

    >>> *===================*
    >>> *2. Continuity correction *
    >>> * ===================*
    >>>
    >>> In studies of rare events, zero events tend to occur and it

    seems common

    >> to

    >>> add a small value so that the zero is taken care of somehow.
    >>>
    >>> If for instance, the inclusion of this small value via continuity
    >>> correction leads to differing results e.g. from

    non-significant results

    >>> when not using correction, to significant results when using

    it, what does

    >>> make of that? Can we trust such results?

    >> If this happens, then the p-value is probably fluctuating

    around 0.05 (or

    >> whatever cutoff is used for declaring results as significant). The
    >> difference between p=.06 and p=.04 is (very very unlikely) to be
    >> significant (Gelman & Stern, 2006). Or, to use the words of

    Rosnow and

    >> Rosenthal (1989): "[...] surely, God loves the .06 nearly as

    much as the

    >> .05".
    >>
    >> Gelman, A., & Stern, H. (2006). The difference between

    "significant" and

    >> "not significant" is not itself statistically significant. American
    >> Statistician, 60(4), 328-331.
    >>
    >> Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the
    >> justification of knowledge in psychological science. American

    Psychologist,

    >> 44, 1276-1284.
    >>

    >>> If one instead opts to calculate a risk difference instead,

    and test that

    >>> for significance, would this be a better solution (more

    reliable result?)

    >>> to the continuity correction problem above?

    >> If one is worried about the use of 'continuity corrections',

    then I think

    >> the more appropriate reaction is to use 'exact likelihood'

    methods (such as

    >> using (mixed-effects) logistic regression models or

    beta-binomial models)

    >> instead of switching to risk differences (nothing wrong with

    the latter,

    >> but risk differences are really a fudamentally different effect

    size

    >> measure compared to risk/odds ratios).
    >>

    >>> Looking forward to hearing your views as diverse as they may

    be in cases

    >>> where there is no consensus.
    >>>
    >>> Sincerely,
    >>> nelly

    >? ? ? ?[[alternative HTML version deleted]]
    >
    > _______________________________________________
    > R-sig-meta-analysis mailing list
    > R-sig-meta-analysis at r-project.org

    <mailto:R-sig-meta-analysis at r-project.org>

    > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

    -- 

    Dr. rer. nat. Gerta R?cker, Dipl.-Math.

    Institute of Medical Biometry and Statistics,
    Faculty of Medicine and Medical Center - University of Freiburg

    Stefan-Meier-Str. 26, D-79104 Freiburg, Germany

    Phone:? ? +49/761/203-6673
    Fax:? ? ? +49/761/203-6680
    Mail: ruecker at imbi.uni-freiburg.de
    <mailto:ruecker at imbi.uni-freiburg.de>
    Homepage:
    https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker