Searching for Good Data about Instructional Efficacy

While vacationing, my husband and son and I were sitting in a local coffee shop, waiting for our pancakes.  I didn’t mean to be listening to the conversation at the next table but the intensity of the young man drew my attention, as he half whispered, “You won’t hear them talk about that publicly.  Look at the building. They have a lot of money.  Only 26% of the elementary kids passed their reading tests though. You won’t hear that talked about.”  

The elderly woman with him (grandmother perhaps?) murmured, “Oh, my. Poor kids. Such a shame.” I didn't get any clear signals that the young man  was a teacher but he was tossing around test scores quite confidently. He mentioned the high school social studies scores which were an anomaly (81%) and attributed them to a particular teacher who, because of the small school size, taught social studies to all students from grades 7–12. His driving concern, though seemed to be the contrast between the “money” which the school apparently “had a lot of” and the test scores which were apparently “shamefully low.”

I always doubt the veracity of numbers that people toss around when they're angry. I wondered where the fellow got his statistics  and ended up googling the school.  I couldn’t easily find the elementary test results but I did find the secondary. In 11th grade, the students were one percentage above the state average  in reading, and twenty points below in writing. Of course the state average isn’t anything to brag about.

So what’s in the data? What does it tell us about the quality of the school and its teachers, how well classroom instruction is addressing the needs and experiences of students, and the effective use of financial resources?

For political reasons, if not for pedagogical, educators pay close attention to the performance of their students on standardized assessments. We have locally elected boards made up of community members with diverse interests and expectations of schools, who typically hold models for evaluating the efficacy of instructional programs that have limited indicators for evaluating success.

Over the course of my career as an educator, I’ve significantly improved my knowledge and skills in developing informal assessment systems in classrooms that help me evaluate student thinking and learning. I’ve also learned how to unpack the goals behind standardized assessments to examine broad patterns in how classes of students respond. I recognize that there is limited utility in how one student responds on a given day to a prompt that may look very different from typical classroom work.

However, it is useful to see the patterns in how groups of students respond to items. Standardized tests are one way to step back and ask how effectively our curriculum is serving students in underrepresented populations vs. Caucasian students, girls vs. boys, or ELL students across the board. It also is informative to find, for example, that in the third grade, comprehension scores are weak,  while fluency is high.  Are we perhaps emphasizing decoding over comprehension instruction in early grades?

One of the most important areas of growth for me professionally was in developing expertise both  in learning how to use standardized data effectively as a tool (which means understanding how and why it is limited in scope and what kinds of uses and interpretations are valid) and in developing multiple ways to assess student learning and thinking in the classroom so that I knew how to better plan instruction for tomorrow and next week.

Below you'll find some Exchange resources related to data usage. But first I’d like to invite you into a conversation where you help identify and develop the resources that we offer.

If you haven’t looked at the NCLE stakeholders, please do so. NCLE comprises 30 stakeholders representing most of the professional organizations in the disciplines, and they're  committed to offering professional service through the Literacy in Learning Exchange  site.

My primary NCLE role is to develop this site so that it offers the professional tools and learning that you need to do your work as an educator effectively. This week, I could use your help in learning more about the following:

  • What tools and materials around using  evidence do you need and would you like this website to provide? Over the next few weeks I’ll be working with  NCLE staff to build our resources around using data effectively (to inform program evaluation, to assist teachers in daily planning, as a tool to inform better policy decision making). I invite you to login and use the comment button to share some ideas or nominate a resource.
  • Do you or your schools have expertise that you would be willing to share (via video, web seminar, vignette, or discussion)? Share your story.
  • Are you in a school team that is getting started with data teams this year? If so, would you be willing to form an Exchange group to make public both your questions and your insights about using data effectively to support student literacy learning?

My request to you is that you talk back and share what you would like to see us develop and provide on this website to inform conversations and actions around using data.

The following resources may be useful to you if you're currently working with this topic, or may give you ideas about additional resources you'd like us to add to this website:

Most Recent: August 30, 2012
  • File Type: Text
  • Sharon Roth

    Creating a narrative around data

    It must be the time of year. I recently had a similar experience of finding myself in a public setting overhearing a conversation about tests scores and the conclusions being reached about the school as a result of it.  In this case it was a family new to town that just received the letter indicating their child's school did not make adequately yearly progress. They received this letter on their child's first day of school. Not having any prior experiences like this they were asking questions just trying to make sense of it all.

     While I was actually part of this group I was the only educator among them. As I tried to share a perspective similar to what KaiLonnie described in her perspective here, I found myself stumbling a bit. I wanted them to understand that the test scores were but one part of the story and leaping to any conclusions about the school on those scores alone was premature at best.  I quickly realized I didn't have the narrative down to help non educators understand the the problems with the conclusions they were reaching.

    Are there any examples out there of schools effectively telling their story to the families and communities they serve?