Confidence interval around random effect variances in place of p-value
This would make an interesting simulation and/or theoretical exercise (I'm going to resist the urge to do it), i.e. identifying the correspondence between p-values constructed from parametric bootstrap full-vs-reduced model comparisons and p-values estimated as fraction of PB fits of full model that give variance=0 for the tested variance component(s).
On 4/2/21 8:22 PM, Jack Solomon wrote:
Well, how about concluding so:
If a (say 2-level) model gives a singular fit (even though perhaps there
is a "tol" that is small but not exactly "0" for that warning to show
up), that would mean we have a "practically" non-significant
random-effect variance component.
On Fri, Apr 2, 2021 at 7:15 PM Ben Bolker <bbolker at gmail.com
<mailto:bbolker at gmail.com>> wrote:
? ? I'm not sure that the bootstrapped CIs *wouldn't* work; they might
return the correct proportion of singular fits ...
On 4/2/21 8:12 PM, Jack Solomon wrote:
> Thank you all?very much. So, I can conclude that a likelihood
ratio test
> and/or a parametric bootstrapping can be used for random effect
variance
> component hypothesis?testing.
>
> But I also concluded that the idea of simply using a bootstrapped
CI for
> a random-effect variance component [e.g., in lme4;
> confint(model,method="boot",oldNames=FALSE)? ] by definition
can't be
> used for significance testing, because it requires the
possibility of
> seeing sd = 0 which can't be "strictly" captured by such a CI from a
> multilevel model (at least not easily so).
>
> I hope my conclusions are correct,
> Thank you all, Jack
>
> On Fri, Apr 2, 2021 at 6:51 PM Ben Bolker <bbolker at gmail.com
<mailto:bbolker at gmail.com>
> <mailto:bbolker at gmail.com <mailto:bbolker at gmail.com>>> wrote:
>
>? ? ? ? Sure. If all you want is p-values, I'd recommend parametric
>? ? ?bootstrapping (implemented in the pbkrtest package) ... that
will avoid
>? ? ?these difficulties.? (I would also make sure that you know
*why* you
>? ? ?want p-values on the random effects ... they have all of the
issues of
>? ? ?regular p-values plus some extras:
>
http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>
>
?<http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>>
>
>? ? ?)
>
>? ? ?On 4/2/21 7:37 PM, Jack Solomon wrote:
>? ? ? > Thanks. Just to make sure, to declare?a statistically
>? ? ?NON-significant
>? ? ? > random effect variance component, the lower?bound of the
CI must be
>? ? ? > EXACTLY "0", right?
>? ? ? >
>? ? ? > Tha is, for example, a CI like: [.0002, .14] is a
>? ? ? > statistically?significant random-effect variance component but
>? ? ?one that
>? ? ? > perhaps borders a p-value of relatively close to but
smaller than
>? ? ?.05,
>? ? ? > right?
>? ? ? >
>? ? ? > On Fri, Apr 2, 2021 at 6:19 PM Ben Bolker
<bbolker at gmail.com <mailto:bbolker at gmail.com>
>? ? ?<mailto:bbolker at gmail.com <mailto:bbolker at gmail.com>>
>? ? ? > <mailto:bbolker at gmail.com <mailto:bbolker at gmail.com>
<mailto:bbolker at gmail.com <mailto:bbolker at gmail.com>>>> wrote:
>? ? ? >
>? ? ? >? ? ? ? ?This seems like a potential can of worms (as
indeed are all
>? ? ? >? ? ?hypothesis tests of null values on a boundary ...)
However,
>? ? ?in this
>? ? ? >? ? ?case
>? ? ? >? ? ?bootstrapping (provided you have resampled appropriately -
>? ? ?you may need
>? ? ? >? ? ?to do hierarchical bootstrapping ...) seems reasonable,
>? ? ?because a null
>? ? ? >? ? ?model would give you singular fits (i.e. estimated
sd=0) half
>? ? ?of the
>? ? ? >? ? ?time ...
>? ? ? >
>? ? ? >? ? ? ? ?Happy to hear more informed opinions.
>? ? ? >
>? ? ? >? ? ?On 4/2/21 6:55 PM, Jack Solomon wrote:
>? ? ? >? ? ? > Dear All,
>? ? ? >? ? ? >
>? ? ? >? ? ? > A colleague of mine suggested that I use the
bootstrapped CIs
>? ? ? >? ? ?around my
>? ? ? >? ? ? > model's random effect variances in place of
p-values for them.
>? ? ? >? ? ? >
>? ? ? >? ? ? > But random effect variances (or sds) start from
"0". So,
>? ? ?to declare a
>? ? ? >? ? ? > statistically NON-significant random effect variance
>? ? ?component, the
>? ? ? >? ? ? > lower bound of the CI must be EXACTLY "0", right?
>? ? ? >? ? ? >
>? ? ? >? ? ? > Thank you very much,
>? ? ? >? ? ? > Jack
>? ? ? >? ? ? >
>? ? ? >? ? ? >? ? ? ?[[alternative HTML version deleted]]
>? ? ? >? ? ? >
>? ? ? >? ? ? > _______________________________________________
>? ? ? >? ? ? > R-sig-mixed-models at r-project.org
<mailto:R-sig-mixed-models at r-project.org>
>? ? ?<mailto:R-sig-mixed-models at r-project.org
<mailto:R-sig-mixed-models at r-project.org>>
>? ? ? >? ? ?<mailto:R-sig-mixed-models at r-project.org
<mailto:R-sig-mixed-models at r-project.org>
>? ? ?<mailto:R-sig-mixed-models at r-project.org
<mailto:R-sig-mixed-models at r-project.org>>> mailing list
>? ? ? >? ? ? >
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models> >? ? ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>> >? ? ? > ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models> >? ? ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>> >? ? ? >? ? ? > >? ? ? > >? ? ? >? ? ?_______________________________________________ >? ? ? > R-sig-mixed-models at r-project.org <mailto:R-sig-mixed-models at r-project.org> >? ? ?<mailto:R-sig-mixed-models at r-project.org <mailto:R-sig-mixed-models at r-project.org>> >? ? ? >? ? ?<mailto:R-sig-mixed-models at r-project.org <mailto:R-sig-mixed-models at r-project.org> >? ? ?<mailto:R-sig-mixed-models at r-project.org <mailto:R-sig-mixed-models at r-project.org>>> mailing list >? ? ? > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models> >? ? ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>> >? ? ? > ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models> >? ? ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>> >? ? ? > >