Best Practices for Comparing TOEIC Speaking Test Scores to Other Assessments and Standards: A Score User’s Guide TOEIC ACTFL CEFR

Author(s):

Schmidgall, Jonathan

Publication Year:

2018

Report Number:

RM-18-05

Source:

ETS Research Memorandum

Document Type:

Report

Page Count:

20

Subject/Key Words:

Test of English for International Communication (TOEIC), American Council on the Teaching of Foreign Languages (ACTFL), Oral Proficiency Interview, Linking and Aligning Scores and Scales, Concordance Tables, Test Scores, Speaking Tests, Common European Framework of Reference (CEFR), Computer Based Tests

Abstract

SUMMARY: In order to better understand the meaning of test scores and to facilitate decision-making, score users may need to understand how scores from two different tests are related. The relationship between scores from two different tests is typically summarized in a concordance table that indicates the correspondence between scores on two tests. Unfortunately, some concordance tables are produced and distributed without any research support, which can lead to inaccurate and unfair decisions about test-takers. First and foremost, score users should know whether a concordance table is backed by research. Lacking such support, a concordance table cannot be assumed to contain information that is trustworthy for making decisions about test takers. This research paper provides a brief, accessible summary of important factors to consider before using a concordance table. The paper explores the relationship between TOEIC Speaking test scores and OPIc test scores, and discusses a problematic concordance table currently being used. The follow-up research paper Linking OPIc Levels to TOEIC Speaking Scores describes a research-based comparison of actual scores from the two tests. ABSTRACT: In order to help score users understand the meaning of test scores, it may be helpful to relate scores on one test to performance levels on another relevant test or set of proficiency level standards. The process of demonstrating a relationship between two tests—or between a test and proficiency level standards—is typically referred to as linking, concordance, or alignment. A variety of research-based methods can be used to carry out this process. When score users rely on concordance tables that are not adequately backed by research, they are likely to make faulty inferences and inappropriate decisions about test takers. Arbitrary or inaccurate concordance may unfairly advantage some and disadvantage others while increasing the likelihood of decision errors. This memorandum provides a brief overview of best practices for (a) linking the scores of two tests and (b) establishing alignment between a test and a set of proficiency level standards. More specifically, it describes how the TOEIC Speaking test and the American Council on the Teaching of Foreign Languages (ACTFL) Speaking standards have been aligned with the Common European Framework of Reference (CEFR) in order to enhance the meaning of test scores. Based on the research that supports these alignments, I hypothesize a relationship between TOEIC Speaking and ACTFL Oral Proficiency Interview computer-based (OPIc) test scores, which is subject to verification. Finally, I highlight a problematic example of a proposed linking of TOEIC Speaking and ACTFL OPIc test scores.

Read More

• Request Copy (specify title and report number, if any)
• https://www.ets.org/Media/Research/pdf/RM-18-05.pdf