NCBIDA - Winter 2005 Newsletter - Connection Between Cognitive Assessment and Teaching Approaches

Connecting Cognitive Assessment and Teaching Approaches
By Nancy Mather & Barbara Wendling

The Northern California Branch of The International Dyslexia Association is privileged to include in our Winter 2005 newsletter an excerpt from the in-press book, Contemporary Intellectual Assessment, edited by Dawn Flanagan et al. Nancy Mather, noted educator and author, has co-authored a chapter with Barbara Wendling entitled "Linking Cognitive Assessment Results to Academic Interventions for Students with Learning Disabilities." Thanks to Chris Voll, NCB-IDA Board member, we have been granted permission to print an excerpt from their work. While general, this excerpt is based on the Cattell-Horn-Carroll (CHC) Theory which documents the range of processing and thinking abilities involved in the learning process. As a result, meaningful evaluation using instruments such as the Woodcock-Johnson III Tests of Cognitive Abilities and Tests of Achievement can now inform planning and recommendations for appropriate and effective remediation and accommodations.

Limitations of global scores

One major problem with the use of intelligence tests for the diagnosis of LD has been an over-reliance on global scores that do not convey useful information for educational planning. A full-scale IQ score simply represents the individual's relative standing compared to the selected norm group, based on his or her performance at a specific point in time on a specific set of tasks designed to measure the test author's conceptualization of intelligence. Stanovich (1999) aptly defined intelligence as "…the statistical amalgamation of a panoply of different cognitive processes" (p. 352). Throughout the century, the limited utility of global ability scores for describing the performance of individuals with LD has been recognized (Kavale & Forness, 1995; Orton, 1925; Reid, Hresko, & Swanson, 1996). As early as 1938, Stern reflected upon the limited value of global IQ scores: "To be sure there has been and there still is exaggerated faith in the power of numbers. For example, 'an intelligence quotient' may be of provisional value as a first crude approximation when the mental level of an individual is sought; but whoever imagines that in determining this quantity he has summed up 'the' intelligence of an individual once and for all, so that he may dispense with the more intensive qualitative study, leaves off where psychology should begin" (p. 60). Unfortunately, in the field of LD, systems of educational classification have been based upon special claims that IQ scores measure intellectual potential, a belief that is neither conceptually nor psychometrically justifiable (Stanovich, 1999).

Tests also vary in the abilities that are assessed and the types of tasks administered then influence the obtained score. If the tasks primarily measure the student's strengths, then a higher score is obtained but if they measure the student's weaknesses, a lower score is obtained. Because intelligence tests often measure certain aspects of the disability or the weak cognitive processes, Orton (1925) cautioned that for individuals with reading problems, full-scale scores provide an entirely erroneous estimate of the person's intellectual capabilities. Similarly, Fletcher et al. (1998) observed that: "To the extent that the child who reads poorly has a significant language disorder, scores on a language-based IQ test will underestimate the child's aptitude for learning in other areas" (p. 200). Composite scores mask the contribution made by reading-related cognitive abilities, so that the correlations and comparisons between these measures must be interpreted with caution (Vellutino, 2001).

The important factor to consider is what abilities are being measured by a certain test and to what aspects of academic performance these abilities are most related. Students with LD often obtain lower scores than normally achieving peers on measures of phonological awareness, rate, memory, and perceptual speed (Gregg et al., in press; Shessel & Reiff, 1999). They will therefore receive lower scores on intelligence tests that place a greater emphasis on these abilities and higher scores on tests that place more emphasis on acquired knowledge and reasoning. Specific abilities also influence performance in different academic areas making some abilities better predictors of certain tasks than others. For example, because they do not require vocabulary or higher-level reasoning, lower-level tasks, such as measures of perceptual speed, are not particularly good predictors of reading comprehension. In contrast, measures of vocabulary and acquired knowledge are good predictors of reading comprehension, but may under-predict or over-predict how well a person will do in math computation.

The major considerations are that content differences among various intelligence and cognitive ability tests result in full-scale score differences, and within these subsets of measures, some are better predictors or more related to certain types of academic performance than others.

Over-reliance on ability/achievement discrepancies

Although intelligence tests are routinely administered to children as one component of a comprehensive evaluation, this practice in many public school settings has been driven more by policy, than diagnostic value. The difficulty in developing a qualitative definition of learning disability, combined with the need to make funding decisions, prompted school districts to use statistical methods to identify students with LD (Silver & Hagin, 2002). For several decades, a common use of a full-scale IQ score has been to predict achievement. Results from an IQ test (usually a global score) were compared directly to the standard scores from achievement tests to determine a discrepancy. Based upon an arbitrary cutting point (e.g., 16 or 22 points), a significant discrepancy was deemed necessary to be eligible for services. The most psychometrically sound procedures employed a correction for the effects of statistical regression.

Much has been written about the limitations of using an IQ/achievement discrepancy for the identification of LD (e.g., Berninger, 2001; Fletcher et al., 2001; Lyon, 1995; Mather & Healey, 1990; Vellutino, 2001), as well as the need to abandon this type of discrepancy (Flowers, Meyer, Lovato, Wood, & Felton, 2000; Fuchs, Mock, Morgan, & Young, 2003). To further complicate matters, state and school district guidelines have varied in regard to the specific method used to define a discrepancy, as well as the magnitude of the discrepancy needed to qualify for services. Therefore, a child may be identified as having a learning disability in one district, but then denied services in another depending on the state and the local criteria or personal philosophy of an independent evaluator who assesses the child (Berninger, 1996). In discussing the discrepancy procedure, Simpson and Buckhalt (1990) stated:

Recent discussions surrounding the reauthorization of the Individuals with Disabilities Education Act (IDEA, 1997; U.S. Department of Education, 2000) as well as the report from the President's Commission on Excellence in Special Education (2002) downplay the need for intelligence testing and advocate a shift away from an IQ/achievement discrepancy as the sole criterion for LD eligibility. If intelligence tests are not needed as part of the eligibility process, most likely their use in public schools will decline. In fact, if a non-categorical approach to services delivery is adopted, as some have recommended, the need for intelligence testing would be reduced or eliminated all together (Tilly, Grimes, & Reschly, 1993). Under this new framework, emphasis would be placed on prereferral interventions and attention would focus on more frequent data collection designed to reveal the effectiveness of educational interventions. LD would be redefined as inadequate response to intervention. Unfortunately, as the consensus grows that the IQ-achievement discrepancy should be abandoned, a valid and validated replacement does not exist (Fuchs et al., 2003).

Even within a response to treatment model, Fuchs et al. (2003) recommend the use of cognitive assessments for students who do not respond to treatments because without their use the LD construct will disappear altogether and lead to a bin called "high-incidence disabilities." For the results of these tests to be useful, however, the focus needs to be on obtaining information that is relevant to behavioral and academic functioning, rather than simply providing an estimate of global ability or calculating an IQ/achievement discrepancy. Even though a student may be deemed ineligible for certain services, all evaluations need to address the referral concerns and propose solutions. As Cruickshank (1977) stated, "Diagnosis must take second place to instruction, and must be a tool of instruction, not an end in itself." The purpose of testing cognitive abilities is to determine the person's unique strengths and weaknesses. In the case of LD, the identified underlying cognitive weaknesses are often directly linked to the specific difficulties in various aspects of school achievement.

Use of profile analysis

The documentation of strengths and weaknesses is often accomplished through profile analysis and consideration of the underlying processes involved in reading, writing, and math. Although identifying an individual's strengths and weaknesses can contribute to educational planning, this practice has been discouraged because of legitimate psychometric concerns. Glutting, McDermott, and Konold (1997) described the various problems with interpreting individual differences through profile analysis, reminding psychologists to "just say no." Although danger does exist in profile analysis when the differences among abilities are minimal and insignificant, significant intra-individual discrepancies among abilities are precisely how intelligence tests can contribute to LD determination and educational planning. As long as interpretation is performed within the context of all data, this type of analysis is supported by the following statement from the Standards for Educational and Psychological Testing (AERA, 1999):

A growing body of research supports the usefulness of factor or specific ability scores in identifying the cognitive processing problems that specifically inhibit school learning (e.g., Evans, Floyd, McGrew, & Leforgee, 2002; Flanagan, Ortiz, Alfonso, & Mascolo, 2002; Gregg et al., in press; Hale, Fiorello, Kavanagh, Hoeppner, & Gaitherer, 2001; McGrew, Flanagan, Keith, & Vanderwood, 1997). As examples, Berninger and Abbott (1994) described oral language and orthographic skills as the best predictors of reading. Adams (1990) found that a child's level of phonemic awareness in kindergarten was the best predictor of reading success in elementary school. Others have described the relationship between slow rapid automatized naming and poor reading skills (e.g., Denckla & Rudel, 1974; Torgesen, 1997; Wolf, Bowers, & Biddle, 2000).

Researchers continue to document the relationships among specific cognitive, linguistic, and academic abilities, identifying prerequisite skills and delineating early indicators of risk. The evolution of theory- and research-based tests measuring multiple abilities has also given professionals the opportunity to gain a better understanding of an individual's unique characteristics. Hannon and Daneman (2001) described the increasing focus on theory: "With the advent and dominance of the information-processing approach to cognition, the emphasis has switched from measurement to theory. The goal is no longer simply to quantify individual differences in intellectual tasks, but also to explain the individual differences in terms of the architecture and processes of the human information-processing system" (p. 103).

Assessments should then focus on understanding a person's information-processing capabilities, including the factors that can facilitate performance. As Gardner (1999) suggested, "We shouldn't ask how smart you are, but rather how are you smart?" Understanding the "constraints" (e.g., limited instruction, specific cognitive or linguistic weaknesses, limited cultural experiences, poor motivation) that impact performance, as well as the multidimensional impact of these constraints (Berninger, 1996), is also important. Because the various constraints affect different aspects of academic functioning, they can help inform the type and extent of accommodations and instruction needed. Interpretation of intra-individual differences and a determination of how these differences affect performance is then the cornerstone for linking the results of cognitive ability tests to meaningful instructional plans.

One basic concept underlying identification of LD is that the difficulties do not extend too far into other domains. In other words, the problem is relatively specific, circumscribed, or domain-specific (Stanovich, 1999). The academic problem is best described as a reading disability, math disability, or spelling disability that is presumably caused by weaknesses in specific cognitive or linguistic processes. The first part of an evaluation is then to determine an initial domain-specific classification (Stanovich, 1999); the next part is to identify the deficient cognitive processes that underlie the disorder (Robinson, Menchetti, & Torgesen, 2002). Learning disabilities are caused by inherent weaknesses in underlying cognitive processes (Robinson et al., 2002). The assessment process can then be viewed as an ability-oriented evaluation designed to help formulate the problem and then determine specific interventions (Fletcher, Taylor, Levin, & Satz, 1995).

- Adams, M. J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press.
- American Educational Research Association (AERA). American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.
- Berninger, V. W. (1996). Reading and writing acquisition: A developmental neuropsychological perspective. Boulder, CO: Westview Press.
- Berninger, V. W. (2001). Understanding the 'lexia' in dyslexia: A multidisciplinary team approach to learning disabilities. Annals of Dyslexia, 51, 23-48.
- Berninger, V. W. & Abbott, R. D. (1994). Redefining learning disabilities: Moving beyond aptitude achievement discrepancies to failure to respond to validated treatment protocols. In G. R. Lyon (Ed.), Frames of reference for the assessment of learning disabilities: New views on measurement issues (pp. 163-183). Baltimore, MD: Paul H. Brookes Publishing Co.
- Carroll, J. B. (1989). Factor analysis since Spearman: Where do we stand? What do we know? In R. Kanfer, P. L. Ackerman, & R. Cudeck (Eds.), Abilities, motivation, and methodology (pp. 43-67). Hillsdale, NJ: Erlbaum.
- Cruickshank, W. M. (1977). Least-restrictive placement: Administrative wishful thinking. Journal of Learning Disabilities, 10, 193-194.
- Denckla, M. B., & Rudel, R. (1974). Rapid automatized naming of pictured objects, colors, letters and numbers by normal children. Cortex, 10, 186-202.
- Evans, J. J., Floyd, R. G., McGrew, K. S., & Leforgee, M. H. (2002). The relations between measures of Cattell-Horn-Carroll (CHC) cognitive abilities and reading achievement during childhood and adolescence, School Psychology Review, 31 (2), 246-262.
- Flanagan, D. P., Ortiz, S. O., Alfonso, V. C., & Mascolo, J. T. (2002). The achievement test desk reference (ATDR): Comprehensive assessment of learning disabilities. Boston, MA: Allyn & Bacon.
- Fletcher, J. M., Francis, D. J., Shaywitz, S. E., Lyon, G. R., Foorman, B. R., Stuebing, K. K., & Shaywitz, B. A. (1998). Intelligent testing and the discrepancy model for children with learning disabilities. Learning Disabilities Research and Practice, 13, 186-203.
- Fletcher, J. M., Lyon, G. R., Barnes, M., Stuebing, K. K., Francis, D. J., Olson, R. K., Shaywitz, S. A., & Shaywitz, B. A. (2001, August). Classification of learning disabilities: An evidence-based evaluation. Paper presented at the U.S. Department of Education LD Summit, Washington, DC.
- Fletcher, J. M., Taylor, H. G., Levin, H. S., & Satz, P. (1995). Neuropsychological and intellectual assessment of children. In H. Kaplan & B. Sadock (Eds.). Comprehensive textbook of psychiatry (pp.581-601). Baltimore, MD: Basic Books, Williams & Wilkens.
- Flowers, L., Meyer, M., Lovato, J., Wood, F., & Felton, R. (2000). Does third grade discrepancy status predict the course of reading development? Annals of Dyslexia, 50, 49-71.
- Fuchs, D., Mock, D., Morgan, P.L., & Young, C. L. (2003). Responsiveness-to intervention: Definitions, evidence, and Implications for the learning disabilities construct. Learning Disabilities Research & Practice, 18, 157-171.
- Gardner, H. (1999). Reframing intelligence. New York: Basic Books.
- Glutting, J. J., McDermott, P. A., & Konold, T. R. (1997). In D. P. Flanagan, J. L. Genshaft, & P. L Harrison (Eds.). Contemporary intellectual assessment: theories, tests, and issues (pp.349-372). New York: Guilford Publications.
- Gregg, N., Jordan, M., Davis, M., Hoy, C., Coleman, C., & Knight, D. (in press). Intelligence measures and Implications for the assessment of learning disabilities at the postsecondary level.
- Hale, J. B., Fiorello, C. A., Kavanagh, J. A., Hoeppner, J. B., & Gaitherer, R. A. (2001). WISC-III predictors of academic achievement for children with learning disabilities: Are global and factor scores comparable? School Psychology Quarterly, 16 (1), 31-35.
- Hannon, B., & Daneman, M. (2001). A new tool for measuring and understanding individual differences in the component processes of reading comprehension. Journal of Educational Psychology, 93(1), 103-128.
- Individuals with Disabilities Education Act Amendments of 1997. (P.L. 105-17). (1997). 20 U.S.C. 1400 et seq. (Congressional Record 1997).
- Kavale, K. A., & Forness, S. R. (1995). The nature of learning disabilities: Critical elements of diagnosis and classification. Hillsdale, NJ: Erlbaum
- Lyon, G. R. (1995). Toward a definition of dyslexia. Annals of Dyslexia, 45, 3-27.
- Mather, N. & Healey, W. C. (1990). Deposing aptitude-achievement discrepancy as the imperial criterion for learning disabilities. Learning Disabilities: A Multidisciplinary Journal, 1(2), 40-48.
- McGrew, K. S., & Flanagan, D. P. (1998). The intelligence test desk reference (ITDR): Gf-Gc cross-battery assessment. Needham Heights, MA: Allyn and Bacon.
- McGrew, K. S., Flanagan, D. P., Keith, T. Z., & Vanderwood, M. (1997). Beyond g: The impact of Gf-Gc specific cognitive abilities research on the future use and interpretation of intelligence tests in the schools. School Psychology Review, 26, 177-189.
- McGrew, K. S., & Woodcock, R. W. (2001). Technical Manual. Woodcock-Johnson III. Itasca, IL: Riverside Publishing.
- Orton, S. T. (1925). Word-blindness in school children. Archives of Neurology and Psychiatry, 14, 581-615.
- President's Commission on Excellence in Special Education. (2002). A new era: Revitalizing special education for children and their families. Retrieved January 5, 2003 from www.ed.gov/inits/commissionsboards/whspecialeducation/reports.html
- Reid, D. K., Hresko, W. P., & Swanson, H. L. (1996). Cognitive approaches to learning disabilities. Austin, TX: PRO-ED.
- Robinson, C. S., Menchetti, B. M., & Torgesen, J. K. (2002). Toward a two-factor theory of one type of mathematics disabilities. Learning Disabilities Research & Practice, 17, 81-90.
- Shessel, I., & Reiff, H. B. (1999). Experiences of adults with learning disabilities: Positive and negative impacts and outcomes. Learning Disability Quarterly, 22, 305-316.
- Silver, A. A. & Hagin, R. A. (2002). Disorders of learning in childhood (2nd ed.). New York: John Wiley & Sons.
- Simpson, R. G., & Buckhalt, J. A. (1990). A non-formula discrepancy model to identify learning disabilities. School Psychology International, 11, 273?279.
- Stanovich, K. E. (1999). The sociopsychometrics of learning disabilities. Journal of Learning Disabilities, 32, 350-361.
- Tilly, W. D., Grimes, J. P., & Reschly, D. J. (1993). Special education system reform: The Iowa story. Communique, 22, 1-4.
- Torgesen, J. K., (1997). The prevention and remediation of reading disabilities: Evaluating what we know from research. Journal of Academic Language Therapy, 1, 11-47.
- U.S. Department of Education. (2000). Twenty-second annual report to Congress on the implementation of the Individuals with Disabilities Education Act. Washington, DC: Author.
- Vellutino, F. R. (2001). Further analysis of the relationship between reading achievement and intelligence: A response to Naglieri. Journal of Learning Disabilities, 34, 306-310.
- Wolf, M., Bowers, P. G., & Biddle, K. (2000). Naming speed processes, timing, and reading: A conceptual review. Journal of Learning Disabilities, 33, 387-407.
- Woodcock, R. W., & Johnson, M. B. (1989). Woodcock-Johnson Psychoeducational Battery-Revised. Itasca, IL: Riverside Publishing.
- Woodcock, R. W., McGrew, K.S., & Mather, N. (2002), Woodcock-Johnson III Tests of Cognitive Abilities and Tests of Achievement, Itasca, IL: Riverside Publishing.

About the Authors

Nancy Mather, Ph.D. is Associate Professor in the Department of Special Education at the University of Arizona and the author of several books about the Woodcock-Johnson.

Barbara J. Wendling has been extensively involved in the training and development related to the Woodcock-Johnson-Revised and the Woodcock-Johnson III. Additionally, she has taught university courses, worked in educational and test publishing, and is currently an educational consultant.

Acknowledgement

Christine Voll graciously helped us obtain permission for this expert. Christine is currently a School Psychologist in the Albany Unified School district; she also maintains a private practice. As a Woodcock-Johnson III and Stanford-Binet Intelligence Scales 5 trainer for Riverside Publishing, she obtained permission for this excerpt from her colleague, Nancy Mather.

This electronic newsletter is a publication of the Northern California Branch of The International Dyslexia Association (NCB-IDA). If you do not wish to receive this newsletter, please send a message to office@dyslexia-ncbida.org with the word "Unsubscribe" in the subject