Meta-analysis on a repeated measures design with multiple trials per subject using metafor

Dear Wolfgang and other readers of the r-help list,

Thank you very much for your suggestion. Unfortunately, the data 
that I have can not be described with a table such as the one you 
have made, because there's no identical trial under both treatment 1 
and treatment 2. To explain, let me explain a bit more about the experiments:

* All subjects were presented with the same number of trials
* Half of these trials were preceded by a prime from category 1 
(treatment 1) and half of these trials with a prime from category 2 
(treatment 2)
* Subjects were asked to respond to these trials (a unique stimulus 
for each trial) by pressing one of two keys on the keyboard.

Because everything was randomized, I can only calculate the total 
number of times a certain response was used under each type of 
trial. There is no pairing of trials under two treatments, so I am 
forced to use the marginal totals from your table.

I have uploaded a simplified version of the data for one experiment 
to illustrate this (the actual experiments have five treatments and 
some have moderators):
https://www.dropbox.com/s/rhgo12cm1asl6x8/exampledata.csv

This is the script that I used to generate the data:
https://www.dropbox.com/s/7uyeaexhnqiiy55/exampledata.R

The problem thus appears to lie mainly in estimating the variance of 
the proportion difference from only the marginal totals, is that 
correct? Is there a way to calculate it from only the marginal totals?

One alternative that I have tried over the last few days, is to use 
the b parameter of interest and it's corresponding standard error 
from the lme4 regression output that I use to analyse the individual 
experiments. Then, I use rma(yi, sei) to do a meta-analysis on these 
parameters. I am not sure this is correct though, since it takes 
into account between-subjects variance (through a random effect for 
subject), and it is sensitive to the covariates/moderators I include 
in the models that I get the b parameters from.

Thanks again for your help, and for any suggestions for solving this problem!

Regards,
Marc


On 07/04/2013 11:21 PM, Viechtbauer Wolfgang (STAT) wrote:

Dear Marc,

Let me see if I understand the type of data you have. You say that 
you have 5 experiments. And within each experiment, you have n 
subjects and for each subject, you have data in the form described 
in your post. Now for each subject, you want to calculate some kind 
of measure that quantifies how much more likely it was that 
subjects gave/chose response 2 under treatment 2 versus treatment 
1. So, you would have n such values. And then you want to pool 
those values over the n subjects within a particular experiment and 
then ultimately over the 5 experiments. Is that correct so far?

Assuming I got this right, let me ask you about those data that you 
have for each subject. In particular, are these paired data? In 
other words, is there are 1:1 relationship between the 30 trials 
under treatment 1 versus treatment 2? Or phrased yet another way, 
can you construct a table like this for every subject:

                 trt 2
              ------------
              resp1 resp2
trt 1 resp1  a     b      10
       resp2  c     d      20
              20    10     30

Note that I added the marginal counts based on your example data, 
but this is not sufficient to reconstruct how often response 1 was 
chosen for the same trial under both treatment 1 and treatment 2 
(cell "a"). And so on for the other 3 cells.

If all of this applies, then essentially you are dealing with 
dependent proportions and you can calculate the difference y = 
(20/30)-(10/30) as you have done. The corresponding sampling 
variance can be estimated with v = var(y) = (a+b)*(c+d)/t^3 + 
(a+c)*(b+d)/t^3 - 2*(a*d/t^3 - b*c/t^3) (where t is the number of 
trials, i.e., 30 in the example above). See, for example, section 
10.1.1. in Agresti (2002) (Categorical data analysis, 2nd ed.).

So, ultimately, you will have n values of y and v for a particular 
experiment and then the same thing for all 5 experiments. You can 
then pool those values with rma(yi, vi) in metafor (yi and vi being 
the vectors of the y and v values). You probably want to add a 
factor to the model that indicates which experiment those values 
came from. So, something like: rma(yi, vi, mods = ~ factor(experiment)).

Well, I hope that I understood your data correctly.

Best,
Wolfgang

--
Wolfgang Viechtbauer, Ph.D., Statistician
Department of Psychiatry and Psychology
School for Mental Health and Neuroscience
Faculty of Health, Medicine, and Life Sciences
Maastricht University, P.O. Box 616 (VIJV1)
6200 MD Maastricht, The Netherlands
+31 (43) 388-4170 | http://www.wvbauer.com

--
Marc Heerdink, MSc. (PhD. candidate)
Dept. of Social Psychology
University of Amsterdam
http://home.medewerker.uva.nl/m.w.heerdink/
http://www.easi-lab.nl/