Message-ID: <a4bc9a8c-8f87-cb1b-423c-08533025f16b@gmail.com>
Date: 2021-04-04T22:58:10Z
From: Ben Bolker
Subject: Confidence interval around random effect variances in place of p-value
In-Reply-To: <CA+sL+8URe+6p4Qd8uZpa_=m3bOffoQvDRbRcfFK6qDMe5XF13Q@mail.gmail.com>
This would make an interesting simulation and/or theoretical exercise
(I'm going to resist the urge to do it), i.e. identifying the
correspondence between p-values constructed from parametric bootstrap
full-vs-reduced model comparisons and p-values estimated as fraction of
PB fits of full model that give variance=0 for the tested variance
component(s).
On 4/2/21 8:22 PM, Jack Solomon wrote:
> Well, how about concluding so:
>
> If a (say 2-level) model gives a singular fit (even though perhaps there
> is a "tol" that is small but not exactly "0" for that warning to show
> up), that would mean we have a "practically" non-significant
> random-effect variance component.
>
>
>
> On Fri, Apr 2, 2021 at 7:15 PM Ben Bolker <bbolker at gmail.com
> <mailto:bbolker at gmail.com>> wrote:
>
> ? ? I'm not sure that the bootstrapped CIs *wouldn't* work; they might
> return the correct proportion of singular fits ...
>
> On 4/2/21 8:12 PM, Jack Solomon wrote:
> > Thank you all?very much. So, I can conclude that a likelihood
> ratio test
> > and/or a parametric bootstrapping can be used for random effect
> variance
> > component hypothesis?testing.
> >
> > But I also concluded that the idea of simply using a bootstrapped
> CI for
> > a random-effect variance component [e.g., in lme4;
> > confint(model,method="boot",oldNames=FALSE)? ] by definition
> can't be
> > used for significance testing, because it requires the
> possibility of
> > seeing sd = 0 which can't be "strictly" captured by such a CI from a
> > multilevel model (at least not easily so).
> >
> > I hope my conclusions are correct,
> > Thank you all, Jack
> >
> > On Fri, Apr 2, 2021 at 6:51 PM Ben Bolker <bbolker at gmail.com
> <mailto:bbolker at gmail.com>
> > <mailto:bbolker at gmail.com <mailto:bbolker at gmail.com>>> wrote:
> >
> >? ? ? ? Sure. If all you want is p-values, I'd recommend parametric
> >? ? ?bootstrapping (implemented in the pbkrtest package) ... that
> will avoid
> >? ? ?these difficulties.? (I would also make sure that you know
> *why* you
> >? ? ?want p-values on the random effects ... they have all of the
> issues of
> >? ? ?regular p-values plus some extras:
> >
> http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects
> <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>
> >
> ?<http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects <http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#testing-significance-of-random-effects>>
> >
> >? ? ?)
> >
> >? ? ?On 4/2/21 7:37 PM, Jack Solomon wrote:
> >? ? ? > Thanks. Just to make sure, to declare?a statistically
> >? ? ?NON-significant
> >? ? ? > random effect variance component, the lower?bound of the
> CI must be
> >? ? ? > EXACTLY "0", right?
> >? ? ? >
> >? ? ? > Tha is, for example, a CI like: [.0002, .14] is a
> >? ? ? > statistically?significant random-effect variance component but
> >? ? ?one that
> >? ? ? > perhaps borders a p-value of relatively close to but
> smaller than
> >? ? ?.05,
> >? ? ? > right?
> >? ? ? >
> >? ? ? > On Fri, Apr 2, 2021 at 6:19 PM Ben Bolker
> <bbolker at gmail.com <mailto:bbolker at gmail.com>
> >? ? ?<mailto:bbolker at gmail.com <mailto:bbolker at gmail.com>>
> >? ? ? > <mailto:bbolker at gmail.com <mailto:bbolker at gmail.com>
> <mailto:bbolker at gmail.com <mailto:bbolker at gmail.com>>>> wrote:
> >? ? ? >
> >? ? ? >? ? ? ? ?This seems like a potential can of worms (as
> indeed are all
> >? ? ? >? ? ?hypothesis tests of null values on a boundary ...)
> However,
> >? ? ?in this
> >? ? ? >? ? ?case
> >? ? ? >? ? ?bootstrapping (provided you have resampled appropriately -
> >? ? ?you may need
> >? ? ? >? ? ?to do hierarchical bootstrapping ...) seems reasonable,
> >? ? ?because a null
> >? ? ? >? ? ?model would give you singular fits (i.e. estimated
> sd=0) half
> >? ? ?of the
> >? ? ? >? ? ?time ...
> >? ? ? >
> >? ? ? >? ? ? ? ?Happy to hear more informed opinions.
> >? ? ? >
> >? ? ? >? ? ?On 4/2/21 6:55 PM, Jack Solomon wrote:
> >? ? ? >? ? ? > Dear All,
> >? ? ? >? ? ? >
> >? ? ? >? ? ? > A colleague of mine suggested that I use the
> bootstrapped CIs
> >? ? ? >? ? ?around my
> >? ? ? >? ? ? > model's random effect variances in place of
> p-values for them.
> >? ? ? >? ? ? >
> >? ? ? >? ? ? > But random effect variances (or sds) start from
> "0". So,
> >? ? ?to declare a
> >? ? ? >? ? ? > statistically NON-significant random effect variance
> >? ? ?component, the
> >? ? ? >? ? ? > lower bound of the CI must be EXACTLY "0", right?
> >? ? ? >? ? ? >
> >? ? ? >? ? ? > Thank you very much,
> >? ? ? >? ? ? > Jack
> >? ? ? >? ? ? >
> >? ? ? >? ? ? >? ? ? ?[[alternative HTML version deleted]]
> >? ? ? >? ? ? >
> >? ? ? >? ? ? > _______________________________________________
> >? ? ? >? ? ? > R-sig-mixed-models at r-project.org
> <mailto:R-sig-mixed-models at r-project.org>
> >? ? ?<mailto:R-sig-mixed-models at r-project.org
> <mailto:R-sig-mixed-models at r-project.org>>
> >? ? ? >? ? ?<mailto:R-sig-mixed-models at r-project.org
> <mailto:R-sig-mixed-models at r-project.org>
> >? ? ?<mailto:R-sig-mixed-models at r-project.org
> <mailto:R-sig-mixed-models at r-project.org>>> mailing list
> >? ? ? >? ? ? >
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> >? ? ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
> >? ? ? >
> ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> >? ? ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>>
> >? ? ? >? ? ? >
> >? ? ? >
> >? ? ? >? ? ?_______________________________________________
> >? ? ? > R-sig-mixed-models at r-project.org
> <mailto:R-sig-mixed-models at r-project.org>
> >? ? ?<mailto:R-sig-mixed-models at r-project.org
> <mailto:R-sig-mixed-models at r-project.org>>
> >? ? ? >? ? ?<mailto:R-sig-mixed-models at r-project.org
> <mailto:R-sig-mixed-models at r-project.org>
> >? ? ?<mailto:R-sig-mixed-models at r-project.org
> <mailto:R-sig-mixed-models at r-project.org>>> mailing list
> >? ? ? > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> >? ? ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>
> >? ? ? >
> ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>
> >? ? ?<https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> <https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models>>>
> >? ? ? >
> >
>