Skip to content

[R-meta] Potential bug in clubSandwich::Wald_test()

5 messages · James Pustejovsky, Stefanou Revesz

Stefanou Revesz
# 
Dear Dr. Pustejovsky (and other Meta Experts),

I've run into a potential bug in clubSandwich::Wald_test().

Would you please confirm whether the error in the following reproducible
example is a bug or not? Any possible fix?

Best,
Stefanou

d <- read.csv("https://raw.githubusercontent.com/fpqq/w/main/f.csv")

fit <- rma.mv(gi ~ teaching_level +pd_format +pd_content +
                      training_hr +time +outcome +area +
                       time_wk_whn +time_wk_btw +study_type,
             V = v_gi,
             random = ~1|study/obs,
             data = d)

Wald_test(fit, constrain_zero(fit$btt), "CR2")

Error in symnum(x$p_val, corr = FALSE, na = FALSE, cutpoints = c(0, 0.001,
 :
  must have 2 'symbols' for logical 'x' argument
James Pustejovsky
# 
I cannot reproduce the error. Here is what I get:
test Fstat df_num df_denom p_val sig
  HTZ     0     15    -0.55    NA

The test you are trying to run is for a configuration where the HTZ test
does not work well. It is a joint test of 15 coefficients but the data
include only 28 distinct studies, so this is a very challenging scenario in
terms of getting the small-sample approximation correct. In particular, the
approximation leads to negative degrees of freedom and a corrected F
statistic of zero because the number of coefficients is so large and the
number of clusters is limited. In this situation, I would suggest using a
cluster wild bootstrap test instead, as described in a recent paper by
Megha Joshi and me:
https://www.jepusto.com/publication/cluster-wild-bootstrap-for-meta-analysis/

The syntax requires only minor modifications (although it takes a bit
longer to compute):
= 20220301)
  Test Adjustment CR_type Statistic    R      p_val
1  CWB        CR0     CR0   Naive-F 1999 0.03951976

James

On Tue, Mar 1, 2022 at 11:15 AM Stefanou Revesz <stefanourevesz at gmail.com>
wrote:

  
  


  


      

    

    

      
      

        


  

        

      Stefanou Revesz
    

    

      
      #
    

  


  

      
Thank you very much, James.  Does changing the `test=` to another option
(e.g., "Naive-F") but keeping the `vcov="CR2"` might still be better than
the QM provided by the rma.mv() model itself?

Wald_test(fit, constrain_zero(fit$btt), "CR2", test = "Naive-F")

Stefanou

        
On Tue, Mar 1, 2022 at 2:49 PM James Pustejovsky <jepusto at gmail.com> wrote:

        

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      James Pustejovsky
    

    

      
      #
    

  


  

      
That approach uses a better variance estimator (CR2) but a very crude
approximation to the distribution of the test statistic. I would guess that
it would still have very poor small-sample properties for contexts such as
your example, where the number of clusters is limited relative to the
number of coefficients being tested.

On Tue, Mar 1, 2022 at 4:00 PM Stefanou Revesz <stefanourevesz at gmail.com>
wrote:

            

      
      
      
    

        

  
  


  


      

    

    

      
      

        


  

        

      Stefanou Revesz
    

    

      
      #
    

  


  

      
For sure, thank you very much.

Stefanou

        
On Tue, Mar 1, 2022 at 5:03 PM James Pustejovsky <jepusto at gmail.com> wrote: