Type-1 error inflation in the traditional by-participant analysis to metamemory accuracy: a generalized mixed-effects model perspective

Murayama, K., Sakaki, M., Yan, V. X. and Smith, G. (2014) Type-1 error inflation in the traditional by-participant analysis to metamemory accuracy: a generalized mixed-effects model perspective. Journal of Experimental Psychology: Learning Memory and Cognition, 40 (5). pp. 1287-1306. ISSN 0278-7393 doi: 10.1037/a0036914

Abstract/Summary

In order to examine metacognitive accuracy (i.e., the relationship between metacognitive judgment and memory performance), researchers often rely on by-participant analysis, where metacognitive accuracy (e.g., resolution, as measured by the gamma coefficient or signal detection measures) is computed for each participant and the computed values are entered into group-level statistical tests such as the t-test. In the current work, we argue that the by-participant analysis, regardless of the accuracy measurements used, would produce a substantial inflation of Type-1 error rates, when a random item effect is present. A mixed-effects model is proposed as a way to effectively address the issue, and our simulation studies examining Type-1 error rates indeed showed superior performance of mixed-effects model analysis as compared to the conventional by-participant analysis. We also present real data applications to illustrate further strengths of mixed-effects model analysis. Our findings imply that caution is needed when using the by-participant analysis, and recommend the mixed-effects model analysis.

Item Type	Article
URI	https://reading-pure-test.eprints-hosting.org/id/eprint/36409
Identification Number/DOI	10.1037/a0036914
Refereed	Yes
Divisions	Central Services Life Sciences > School of Psychology and Clinical Language Sciences > Department of Psychology
Publisher Statement	This article may not exactly replicate the final version published in the APA journal. It is not the copy of record
Download/View statistics	View download statistics for this item