Test-Retest Reliability-The Good, the Bad, and the Judgment Calls!
An Ode to Retesting Test, test, and test again.
Wait, wait a minute-does it matter WHEN?
Is it weeks or months, days or years?
Do I have enough info to tell all my peers?
What will happen to the students’ scores?
Will kids remember the answers or run out the doors?
Will the scores be valid, reliable, and true?
Will I interpret correctly so no one will sue?
Oh, test publisher, hear my refrain;
Give me some guidance, it’s taxing my brain!!
One should begin any dry but important topic with a bit of levity…hence the original poem above (inspired by Dr. Seuss, of course)! Welcome to the second issue of the Clinical Cafe, with today’s “espresso shot” topic of test-retest reliability. We, in Development for Pearson’s Assessment group, get numerous calls each month from professionals around the country regarding rules and strategies for retesting students. So, here’s another “insight” to read with your morning (or afternoon or evening) coffee and share with your colleagues.
Overall, test-retest reliability is an index of temporal stability. It tells how much the individual’s normative score might possibly change on retesting if a period of time has elapsed between test administrations. Change could reflect the person’s growth or fluctuation in the ability being measured, random differences in performance, or the individual’s recollection of the earlier administration. A test-retest coefficient is a statistical measure that is obtained by administering the same test twice, with a certain amount of time between administrations, and then correlating the two score sets. Reliability between two parallel forms with different content is known as an alternate-form coefficient (as in the PPVT-III).
When making a decision on retesting, follow the steps below.
- Determine why you are conducting a retest: Did the examinee’s performance fall below your expectations due to illness, bad day, test anxiety, student behavior, etc? If this is the case, as soon as the examinee is up to it, you should be able to retest, especially if you use a parallel form. Are you involved in a pre/post test situation where you are attempting to ascertain gain? If so, you’d want to schedule the second administration after the completion of instruction or therapy to determine the effectiveness of the treatment.Has the student recently transferred from a different school? If the original test administration was done in another school or setting by a clinician that you do not know, and you question the reported results, you may choose to re-test.
- Locate the section in each of the Pearson test manuals that discuss test-retest reliability:
TEST Manual Page(s) Test-Retest Time Interval Test-Retest Median Interval Reliability Coefficients CASL 121-124 7-109 days 6 weeks .65-.96 EVT 67-69 8-203 days 42 days .77-.90 GFTA-2* 52-54 0-34 days 14 days .79-1.00 KLPA-2* 66-67 0-34 days 14 days .79-1.00 OWLS LC/OE 123-125 20-165 days 8 weeks .73-.89 OWLS WE 146-147 18-165 days 9 weeks .87-.90 PPVT-III Form A 48-51 One month 30 days .91-.93 PPVT-III Form B 48-51 One month 30 days .91-.94 PPVT-III A & B** 48-51 0-7 Same day .88-.95
*Read further for an additional issue related to test content.
- **The PPVT-III is unique among Pearson speech-language tests in that it has parallel forms (Forms A & B). As written in the Alternate-Forms Reliability Coefficients section on page 48 of the PPVT-III manual, most test subjects were given both forms of the test in the same day.
- Make a clinical decision based on all the information:While some tests provide clinicians with an exact test-retest waiting period, some do not. So much depends on the reason for the retesting. After reviewing the information provided in the manual and following the above steps, you may rely on your professional clinical judgment to determine when to confidently retest. You can be confident of the test reliability if your retesting falls within the respective test interval in the above table since you will be matching standardization procedures.
Test Content and Test-Retest Reliability: A Related Consideration
Test content also should be considered when determining the need for retesting. For example, articulation is a developmental skill area; you would expect more of a change in performance earlier on the growth curve (i.e., ages 2-5), which then will flatten and stabilize around 8 years of age. Retesting a young student at a given time interval on the GFTA-2 and/or the KLPA-2 would likely have greater change attributed to growth than retesting a student at 10 years of age. This is one reason that the test-retest time interval is narrower for these two tests.In language, skills that are “closed set” (e.g., syntax) should be considered differently than those that are “open set” (e.g., vocabulary, nonliteral language). Vocabulary has a steep growth curve similar to articulation, but does not flatten at a particular age; it grows throughout the life cycle. Conversely, syntax forms are a closed set; you learn them early, and they are generally static. So it helps to make the test-retest decision-making and data analysis seem clearer when you take into account test content.
Dancing the Line of Clinical Judgment and Test Standardization
Yes, it is difficult to know sometimes what you can and can’t make decisions on. Test standardization is a rigorous process and we appreciate professionals being concerned about “following the rules.” At the same time, no test can anticipate all the situations and nuances of the clinical arena, so there is a point at which the rules end and your clinical judgment begins. We’re happy to continue to help you clarify the line on which to “dance.”If you want more information on standardized test development, the Development Team at Pearson’s Assessment group is in the process of creating an ASHA-approved CE presentation (available end of summer) on the basics of assessment. If you are interested in scheduling a continuing education activity for your school, district, or your state speech-language-hearing association, please contact:
As always, we’d like to thank you for your ongoing service to people with communication needs, and we are here to support you with that effort. If you’d like to discuss this topic further, please feel free to use the SLP Forum as the vehicle for an ongoing discussion with your colleagues. Should you have questions regarding these or other Pearson Speech and Language products, we welcome your phone calls at 800-627-7271 or use our web site at http://psychcorp.pearsonassessments.com. Oh, yes, and if you’d like to copy the poem at the top, feel free (but don’t forget to cite the Clinical Café)!
Enjoy the summer!