September 21, 2010
NASHVILLE, Tenn.—Rewarding teachers with bonus pay, in the absence of any other support programs, does not raise student test scores, according to a new study issued today by the National Center on Performance Incentives at Vanderbilt University’s Peabody College of education and human development in partnership with the RAND Corporation.

This and other findings from a three-year experiment – the first scientific study of performance pay ever conducted in the United States – were released at a conference on evaluating and rewarding educator effectiveness hosted by the National Center on Performance Incentives at Vanderbilt.

Paying teachers bonuses based on their performance has been a controversial issue nationwide since the 1950s, but until now the concept has never been scientifically researched.

“We tested the most basic and foundational question related to performance incentives — Does bonus pay alone improve student outcomes? – and we found that it does not,” Matthew Springer, executive director of the National Center on Performance Incentives, said. “These findings should raise the level of the debate to test more nuanced solutions, many of which are being implemented now across the country, to reform teacher compensation and improve student achievement.”

The Project on Incentives in Teaching, called the POINT Experiment, took place over the 2007 – 2009 school years with participation by mathematics teachers in grades 5 through 8 in Metropolitan Nashville Public Schools. Nearly 300 teachers, approximately 70 percent of all middle-school math teachers in Nashville’s public schools, volunteered to participate. The complete study, including setup and analysis, began in 2005 and ended in 2010.

POINT tested no other types of incentives or systems of support for the teachers, such as professional development or guidance on instructional practices – many of which have evolved over the five years since POINT began.

“We designed POINT in this manner not because we believed that an incentive system of this type is the most effective way to improve teaching performance, but because the idea of rewarding teachers on the basis of students’ test scores has gained such currency,” Springer said. “We sought a clean test of the basic proposition: If teachers know they will be rewarded for an increase in their students’ test scores, will test scores go up? We found that the answer to that question is no. That by no means implies that some other incentive plan would not be successful.”

Here’s how the POINT experiment worked:

• Following a year of detailed project design by a multi-disciplinary team from Vanderbilt and RAND, all middle-school math teachers in Nashville were invited to volunteer for the experiment. Approximately 70 percent of all middle-school math teachers in Nashville’s public schools stepped forward to participate.

• Approximately half of the nearly 300 volunteers were randomly assigned to a “treatment” group, in which they were eligible for bonuses of up to $15,000 per year on the basis of their students’ test-score gains on the Tennessee Comprehensive Assessment Program (TCAP).

• The other half were assigned to a “control” group not eligible for these bonuses. Teachers were evaluated based on an historical performance benchmark for MNPS teachers, not on competition with one another. All teachers in the treatment group had the chance to earn bonuses. (The names of participating teachers – and which group they were in – have been kept confidential by the research team.)

• The annual bonus amounts were $5,000, $10,000 or $15,000. Over the course of the experiment, POINT paid out more than $1.27 million in bonuses. Overall, 33.6 percent of the original group received bonuses, with the average bonus being approximately $10,000.

Teacher attrition occurred during the experiment. About half of the 296 teachers who initially volunteered remained through the end of the third year. The teachers who left the study either left the school system, moved to other grades or stopped teaching mathematics. Only one participating teacher specifically asked to be removed from the experiment.

While there was no overall effect on student achievement across the entire treatment group, the researchers found a significant benefit for fifth graders in Year 2 and Year 3 of the experiment: fifth graders taught by teachers who earned bonuses did show gains in test scores. However, the effect did not carry over to sixth grade when students were tested the following year. Springer said this finding raises questions about what is different about fifth grade and what factors –student development, curriculum, teaching and classroom structure – may have played a role.

He also noted that implementation of POINT went smoothly, with no complaints from teachers about the calculation of bonuses, the payment of awards, bonuses they did or did not receive or the fairness of the process. This in itself is a significant finding, Springer said, because historically, teacher associations have opposed performance or merit pay plans, particularly if the pay plan awarded teachers solely on their individual value-added score.

Springer attributed this smooth implementation of the POINT experiment to a broad partnership involving the Metropolitan Nashville School Board and Metropolitan Nashville Public Schools administrators, the Mayor’s Office, the Metropolitan Nashville Education Association, the Nashville Alliance for Public Education, the Tennessee Education Association and the Tennessee Department of Education. The POINT experiment team received guidance and support from these organizations, as well as the participating teachers, throughout the project.

“We believe there is an important lesson here: teachers are more likely to cooperate with a performance pay plan if its purpose is to determine whether the policy is a sound idea, than with plans being forced on them in the absence of such evidence and in the face of their skepticism and misgivings,” Springer said.

Click here to read the full report and watch a video of the announcement.The National Center on Performance Incentives at Vanderbilt University’s Peabody College of education and human development partnered with the RAND Corporation, a nonprofit public policy research institute based in Santa Monica, Calif., in 2006 to complete the study. The POINT experiment was funded by the U.S. Department of Education’s Institute of Education Sciences. Performance bonuses were funded by a private donor.

Springer is an assistant professor of leadership, policy and organizations at Peabody College. For more information about Peabody College, ranked the No. 1 education school in the nation by U.S. News & World Report for the past two years, visit http://peabody.vanderbilt.edu/. For more Vanderbilt news, http://news.vanderbilt.edu/.