Hope this finds everyone well,
I was hoping to get some feedback on addressing some practical concerns
with an idea for a meta-analysis that I have. At the core, I'm concerned
with the interpretability of the data analysis, so I'm trying to check
whether this is even possible before I start the study.
My field commonly administers lists of words that a participant must read
correctly. Item difficulty is often summarized in studies as the proportion
of respondents who got an item correct over the total sample size. Studies
use different word-reading tests and lists of varying length (usually
between 20 and 50). Each test differs in how long the words are
presented, how many words are presented at a time, etc. I'm interested in
whether certain word-specific variables (e.g., frequency of word in
English) are related to the item difficulty (i.e., proportion of people who
answer it correctly).
My current conceptualization of the problem is as a multilevel
meta-regression. The effect size is computed from the proportion of correct
answers to each word which is nested within study and within tests. I would
have multiple covariates (i.e., word traits) for each effect size,
covariates at the study level (i.e., average sample demographics), and
covariates at the test level (i.e., administration differences). My concern
is that these are too many predictors to include in a meta-regression, and
I also am anticipating that I won't have any information about
the covariance matrix of the effect sizes. Any recommendations, words of
warning, caveats, or suggestions would be incredibly helpful since I'm
early in the process.
Thank you!
[[alternative HTML version deleted]]