[R-meta] confidence interval very large
That's a good question. No, the same concerns apply to CIs and to SEs and t-statistics. For F-statistics our research (Tipton & Pustejovsky, 2015 in JEBS) found that F-tests tend to be conservative, with lower-than-nominal Type-I error, when df are very low. So they're not necessarily problematic in terms of Type I error control, but they do tend to have very low power so still wouldn't put much stock in them.
On Wed, Jul 27, 2022 at 3:41 PM Huang <wuhuang0421 at gmail.com> wrote:
Thanks so much for your responses. Dr.Pustejovsky, besides confidence interval, are other results (e.g., robust SE and t-tests & F-tests trustable) when df < 4? I appreciate your help. Best wishes, Huang On Wed, Jul 27, 2022 at 11:51 AM James Pustejovsky <jepusto at gmail.com> wrote:
Hi Huang, The wide confidence interval is due to having only one degree of freedom (a t distribution with 1 degree of freedom is equivalent to the dreaded, ill-behaved Cauchy distribution). I would not trust confidence intervals based on so few degrees of freedom. Tipton (2015 Psych Methods) suggests disregarding results based on RVE if df < 4. Generally, low degrees of freedom means that there is very little information available to estimate standard errors--i.e., the SEs will be very noisy. This can happen, for instance, if you are comparing two different categories of effect sizes and one of the categories only appears in two studies. You could, in these instances, report the model-based confidence interval, but in doing so you're relying on all of the additional assumptions of the model (such as homoskedasticity of random effects, correctness of correlations between sampling errors). So you would need to be clear about these caveats. You might consider also reporting sensitivity analyses under some range of plausible alternative assumptions. James On Tue, Jul 26, 2022 at 10:39 PM Huang <wuhuang0421 at gmail.com> wrote:
Hi folks, I am using conf_int to calculate the confidence interval but the results showed the confidence interval are very large. For example, I have a mean estimate 0.1655 with a SE of 0.3014 (df = 1), but the 95% confidence interval is [-3.659, 3.990]. Is this correct? Should I report the model based confidence interval instead of RVE based? Thank you so much for taking time to respond to my email. Best wishes, HUang
[[alternative HTML version deleted]]
_______________________________________________ R-sig-meta-analysis mailing list @ R-sig-meta-analysis at r-project.org To manage your subscription to this mailing list, go to: https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis