Skip to content

[R-meta] confidence interval very large

5 messages · Guido Schwarzer, Huang Wu, James Pustejovsky

Original
Huang Wu
# 
Hi folks,

I am using conf_int to calculate the confidence interval but the results
showed the confidence interval are very large. For example, I have a mean
estimate 0.1655 with a SE of 0.3014 (df = 1), but the 95% confidence
interval is [-3.659, 3.990]. Is this correct? Should I report the model
based confidence interval instead of RVE based? Thank you so much for
taking time to respond to my email.

Best wishes,
HUang

  
  


  


    

    

      
      

        


  

        

      Guido Schwarzer
    

    

      
      #
    

  


  

      
Am 27.07.22 um 05:39 schrieb Huang:

            

      
      
      
    

        
Looks like it is. I get similar values calculating the CI using base R:

0.1655 + c(-1, 1) * 0.3014 * qt(0.975, df = 1)

Best,

Guido

  
  


  


      

    

    

      
      

        


  

        

      James Pustejovsky
    

    

      
      #
    

  


  

      
Hi Huang,

The wide confidence interval is due to having only one degree of freedom (a
t distribution with 1 degree of freedom is equivalent to the dreaded,
ill-behaved Cauchy distribution). I would not trust confidence intervals
based on so few degrees of freedom. Tipton (2015 Psych Methods) suggests
disregarding results based on RVE if df < 4.

Generally, low degrees of freedom means that there is very little
information available to estimate standard errors--i.e., the SEs will be
very noisy. This can happen, for instance, if you are comparing two
different categories of effect sizes and one of the categories only appears
in two studies.

You could, in these instances, report the model-based confidence interval,
but in doing so you're relying on all of the additional assumptions of the
model (such as homoskedasticity of random effects, correctness of
correlations between sampling errors). So you would need to be clear about
these caveats. You might consider also reporting sensitivity analyses under
some range of plausible alternative assumptions.

James

        
On Tue, Jul 26, 2022 at 10:39 PM Huang <wuhuang0421 at gmail.com> wrote:

        

            

      
      
      
    

        

      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Huang Wu
    

    

      
      #
    

  


  

      
Thanks so much for your responses. Dr.Pustejovsky, besides confidence
interval, are other results (e.g., robust SE and t-tests & F-tests
trustable) when df < 4? I appreciate your help.

Best wishes,
Huang

On Wed, Jul 27, 2022 at 11:51 AM James Pustejovsky <jepusto at gmail.com>
wrote:

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      James Pustejovsky
    

    

      
      #
    

  


  

      
That's a good question. No, the same concerns apply to CIs and to SEs and
t-statistics.

For F-statistics our research (Tipton & Pustejovsky, 2015 in JEBS) found
that F-tests tend to be conservative, with lower-than-nominal Type-I error,
when df are very low. So they're not necessarily problematic in terms of
Type I error control, but they do tend to have very low power so still
wouldn't put much stock in them.

        
On Wed, Jul 27, 2022 at 3:41 PM Huang <wuhuang0421 at gmail.com> wrote: