What Does Scientifically Based Research Mean for Schools?

The No Child Left Behind Act has increased pressure on schools to identify programs that improve student achievement. The law requires schools to adopt new programs based on rigorous research that proves they are effective. Supporters say requiring schools and districts to adopt such programs takes the guesswork out of what works. Detractors say the federal government's mandate is too strict and costly. Whichever side you are on, schools that fail to see gains in student achievement will face tough consequences under No Child Left Behind (NCLB). As educators review programs with a proven track record of success, they are sifting through evidence with an eye toward research that will meet the federal government's standards.

What Qualifies as Scientifically Based Research?

The U.S. Department of Education says scientifically based research applies rigorous, systematic, and objective procedures to evaluate whether a program is effective.

Russ Whitehurst, who heads the Institute of Education Sciences of the U.S. Department of Education, says requiring schools to adopt programs backed by scientific evidence is new to education. Until now, he says too many schools have adopted programs based on hunches and anecdotes. "People want to make wiser choices," Whitehurst says. "They don't want to gamble. This generates an appetite for more information on what will work best."

The U.S. Department of Education backs research employing randomized, controlled trials that assign subjects to an experimental group or a comparison group to test a program's effectiveness— an approach commonly used in medicine, but less often in education.

Some researchers caution that randomized trials can be limiting. "A randomized study invariably simplifies the world," says professor David Berliner of Arizona State University's College of Education in Tempe. "You can only look at five or six variables at a time. In the real world, there are many more factors." Berliner argues the federal government should not promote a single method of research. "The real question is what constitutes science? What will you accept as evidence?"

Many others, like principal Scott Steckler of George Cox Elementary School in Gretna, Louisiana, worry the new mandate will prove too costly, especially when schools and states are strapped for cash. "The people promoting No Child Left Behind are of the opinion that we can do the job with the same amount of money. That's ludicrous." NCLB supporters maintain it is not necessarily more expensive to implement programs and practices rooted in scientifically based research, especially since proven programs are likely to be more effective, resulting in less waste in the long run. Steckler sees other benefits to the scientifically based research requirements, however. Such requirements "could eliminate some shady practices," he says. "I have seen programs adopted because the superintendent was well acquainted with the chief sales rep of the company."

The U.S. Department of Education has called randomized studies the "gold standard" in research and is earmarking $47 million for such trials in early reading instruction, alternative certification of teachers, English language learners, charter schools, and several other areas. While randomized trials are heavily emphasized, Whitehurst points out that this approach is one of several accepted under the law. Other research methods include quasi-experimental studies, rigorous data analysis, and observational methods.

What Programs Are Affected?

The call for scientifically based research in education casts a wide net, affecting not only academic programs, like reading and math, but efforts to curb drug use, keep schools safe, increase parental involvement, and improve professional development for teachers.

What's less clear is how aggressive educators must be to ensure that all new programs meet the tough new standards. Even the U.S. Department of Education's top researcher acknowledges there's "ambiguity" on this question.

"Most people would agree that before a state adopts a reading program, you would want a high degree of evidence showing the program is effective," Whitehurst says. "But when it comes to supplemental materials, such as workbooks, do we expect they will be as closely evaluated? Probably not."

Some states, like Arkansas and Louisiana, already require schools to adopt new programs backed by evidence. "I don't know that we'll face a lot of challenges," says principal Gerald LeBlanc of Homedale Elementary School in Harvey, Louisiana, noting that such programs are "not new to us."

While the U.S. Department of Education has outlined standards for scientifically based research, comprehensive data-collection efforts by schools, districts, and states can also count as scientifically based research.

Arkansas was one of the first states to receive federal funding for the Arkansas Reading First program under the new mandate. Ray Simon, who heads the Arkansas Department of Education and is a recent nominee for the position of assistant secretary for the Office of Elementary and Secondary Education, says the state's successful funding was in part due to the Arkansas comprehensive student achievement database. The data showed that reading scores increased and the program worked, especially in kindergarten through fourth grade. Simon says the extensive data-collection efforts met the tough standards set forth in NCLB.

"I consider that research-based," he says." How are certain groups of children doing? Educators can find that out." Still, Simon acknowledges some schools in his state will struggle to evaluate research and data to determine if they meet the criteria set forth by the federal government. He's encouraging schools to work closely with local educational cooperatives and his agency. "There are vendors all over claiming they can do this and do that but we want to make sure the program works," he says.

Help for Educators

While supporters believe the call for rigorous research will improve the quality of learning, they wonder whether educators have the time and expertise to conduct a thorough review of research. "We do have lists of what we mean by research-based programs, but just having the time to look at them, the funds to afford them, and the buy-in from teachers will be our biggest challenges," says Imelda Guerra, a SEDL board member and the principal of Magee Elementary School in Alice, Texas.

We can create a culture in schools that places an emphasis on scientifically based research and gives principals the tools to judge.

Russ Whitehurst, Institute of Education Sciences

Some district administrators are helping their principals and teachers become savvier consumers of commercial and homegrown programs. Gloria Griffin, another SEDL board member who is superintendent of the Millwood Public School District in Oklahoma City, is tapping federal dollars to give teachers more time to use data in their analysis of student work during the summer and on Saturdays. "We are using our training dollars to enhance preparation of teachers coming out of school and to help those who have been in the field a long time," she says.

At Homedale Elementary, LeBlanc has convened a team of teachers that regularly reviews research to see which newly proposed programs are scientifically proven. He encourages them to visit other schools to see if the program is really getting the results claimed by the research. Teachers also field-test the program to see if it works with the school's student population before adopting it. And the team digs up as many research studies as possible on the program, looking for contradictions.

Others are turning to consultants for help, but they acknowledge that outside expertise is often costly. "I don't expect that we are going to get to a point where the typical school principal is trained as a researcher," Whitehurst says. "But we can create a culture in schools that places an emphasis on scientifically based research and gives principals the tools to judge."

Why Evidence Matters
An Interview with Russ Whitehurst, Director, Institute of Education Sciences,
U.S. Department of Education

How is this new approach different from what schools and state education agencies have done in the past?
The conversation heretofore would have been a conversation that was driven by anecdotes or by suggestions: "If you want to find out how good this is, call superintendent so-and-so and he will tell you what a good experience he has with it in his schools."

The way that interaction will occur a year or two from now is that the vendor will be asked by the superintendent, "What evidence do you have of the effectiveness of this program for kids like ours?" The vendor may respond, "We do have the evidence." The superintendent may ask, "Has it been vetted by the What Works Clearinghouse?" If the answer is yes, the vendor will be told to call back in a couple of weeks and the school will check it out. If the vendor says no, they'll be asked, "Why not?"

How do you respond to critics who say scientifically based research is not clearly defined?
Well, I think with respect to what works . . . the definitions are quite clear. We know what works through randomized trials. The missing piece—and the piece we intend to provide with the What Works Clearinghouse—is a place where practitioners can turn for impartial, carefully vetted research information.

I also think there's confusion about when we need rigor and when we don't and that's the nature of the enterprise.

What advice would you give to districts and schools who want to ensure that new programs are scientifically based?
First, see if the What Works Clearinghouse has issued an evidence report in the area in which you are making a decision. I also think that people making high-level decisions about programs and practices need to become more aware about what the rules of evidence are. Districts and schools also need to develop their own measures to see if they are meeting their performance goals.

Some critics argue randomized trials take too long and are too expensive. Your reaction?
If it's over the course of a school year, you can do it over the school year plus a summer. There's nothing inherently time-consuming about randomized trials. What is the expense of not knowing what works? The cost of finding out what works is small compared to making bad decisions.

Others argue the federal government is promoting a particular view of education.
I'm not sure what that means. What view? The only view it is promoting is that we need to use evidence to determine what works and why. The view of education that may be ruled out is that education is an art and it will never be more than accumulated craft wisdom. These are very pessimistic views of education.

How will you enforce this provision?
The accountability provisions in No Child Left Behind are very real. Schools are held accountable for progress. When they apply for funding, applicants are required to propose programs based on evidence that a program a school is planning to implement has evidence of effectiveness.

What impact will this provision have on public schools in the long run?
The thing that drives me every morning when I wake up is that this might actually make a difference and we will start to see impressive gains. We'll get to the point in the not too distant future where every child will be guaranteed an education that is good enough for that child's future.

What Works Clearinghouse Priorities

The clearinghouse will conduct comprehensive research reviews in these areas:

• Interventions for beginning reading
• Curriculum-based interventions for increasing K-12 math achievement
• High school dropout prevention
• Peer-assisted learning in elementary schools
• Programs for increasing adult literacy
• Interventions to reduce delinquent, disorderly, and violent behavior in and out of school n Interventions for elementary English language learners

What Works Clearinghouse

To help educators navigate the research on intervention strategies, the U.S. Department of Education has created the What Works Clearinghouse, which will provide an independent source of information on what works in education. That information will be based on a rigorous review of existing research.

The clearinghouse will publish evidence reports that review research evaluating the effectiveness of programs, products, practices, and policies on its Web site at http://ies.ed.gov/ncee/wwc/. "We're not in the business of endorsing products," says clearinghouse spokesman Steve Fleischman. "We're providing what has never existed before: a set of highly credible research review tools that will be applied consistently to judge the evidence of effectiveness across all kinds of things."

For now, the clearinghouse is focusing its work on several key areas including reading, math, dropout prevention, and school safety. Fleischman says educators can nominate topics and intervention strategies online for consideration.

The What Works Clearinghouse also offers names of individuals and organizations that can evaluate intervention strategies for states, districts, and schools, using the standards approved by the U.S. Department of Education.

For example, a large district may have the need to hire an evaluator to conduct a randomized, controlled trial of a social studies program yet to be reviewed by the clearinghouse's advisory group. Or they could tap experts to evaluate a homegrown program before it's adopted at the district or state level.

"Typically, you find a researcher by word of mouth or at a conference," Fleischman says. "But for the first time, educators will now have ready access to evaluators who make the claims that they can provide these services." He says the clearinghouse will not make judgments about the evaluators' qualifications.

What's Next?

While some educators are confident they can meet the federal government's new standards, others aren't so sure. They worry that the definition of scientifically based research is too narrow, leaving behind promising practices and programs. They also question whether randomized, controlled studies can be conducted in an environment as dynamic as public education. Still others believe the emphasis on scientifically based research is good for education and long overdue.

Individuals on both sides of the debate will be watching closely to see if the mandate results in improved student learning and achievement.

Lesley Dahlkemper is the president of Denver-based Gracie Communications, Inc., a firm specializing in K-12 education writing, communications strategy, and project management.

Next Article: State Efforts Push to Leave No Child Behind in Mathematics and Science