Frequently Asked Questions…About Our Work in Metropolitan Nashville Public Schools

Participate

Q

Who is eligible to participate?

A

To participate in the experiment, teachers have to instruct mathematics in grades 5, 6, 7, or 8 with ten or more students in mathematics expecting to take the TCAP at the end of the year. Teachers do not have to be full-time math teachers; in fact, teachers are allowed to have students for other subjects, such as science or English. Additionally, a teacher’s math students can be a mix of different subjects and grades.

Q

How will the experiment work?

A

Participation in this experiment is entirely voluntary for teachers. Approximately 200 volunteers were needed to initiate this experiment. Teachers were randomly assigned to one of two groups: a treatment group and a control group. Teachers in the treatment group are eligible to earn bonuses based on their students’ academic progress during the school year, as measured by their gains on the TCAP. Control group teachers are not eligible for bonuses. However, all participating teachers, in both the treatment and control group, will receive stipends of $750 each year for their participation in the experiment.

Q

How much extra work will I have to do?

A

NCPI asks participants, in both the treatment and control group, to complete one or two surveys each year. These surveys are not expected to take more than 30 minutes to complete. Some teachers may also be asked to conduct an in-person interview with researchers. Any other activities such as taking steps to improve instructional practice are up to the participating teacher. NCPI does not require teachers to adopt particular teaching methods, to attend workshops, or to complete written reports.

Q

How long does the experiment last?

A

The experiment is planned to last three years. Teachers assigned to the treatment group remain in that group for the three year duration. Likewise, teachers assigned to the control group remain in that group for three years.

Q

If I do not want to continue in the experiment, may I stop?

A

Teachers are allowed to leave the experiment at any time by notifying NCPI that they no longer wish to participate. However, if a teacher leaves before the end of the year, he/she forfeits the annual stipend of $750.

Q

If I do not participate in the first year, will I have another chance to sign up later?

A

NCPI anticipates that some participating teachers will move out of the area or otherwise leave their jobs before the end of the experiment. If this happens, NCPI will attempt to replace them. First priority is given to teachers who volunteered to participate in year one, but were placed on a waiting list due to an excess number of volunteers. If additional replacements are needed, NCPI will call for new volunteers in the district.

Q

If I am assigned to the control group, will I have a chance to switch to the treatment group later?

A

No, teacher assignment to the treatment or control group is permanent.

Q

If I change schools, will I remain in the experiment?

A

Teachers remain in the experiment as long as they continue to teach mathematics at the middle school level in Metro Nashville Public Schools.

Q

Will identities of participants be made public? Who will know I earn a bonus?

A

The National Center on Performance Incentives will not make public the names of participants. The district has also agreed to preserve confidentiality of the names of participants and the teachers who earn bonuses. Obviously, someone in the district administration must have access to the names of the winners in order to send checks to the right individuals. However, this individual will not make this information public.

In addition, we asked all participating teachers to sign a pledge of confidentiality not to reveal to other district employees whether they have been assigned to the treatment or the control group. In addition to protecting the integrity of the experiment, this provides teachers with any easy way to deflect prying questions. All teachers need to say if asked whether they are eligible for a bonus is that they have signed a pledge of confidentiality.

Q

Why should assignment to treatment and control groups be kept confidential?

A

As in any experiment, treatment and control groups should be as much alike as possible, apart from the fact that one group receives the treatment and the other does not. Confidentiality helps to ensure that all teachers in the experiment will be treated normally by parents, by principals, and by colleagues, and keeps unwanted influences out of the experiment.

Q

Will my performance in the experiment be used to evaluate me? Will my performance be compared to other participating teachers?

A

No.

Q

Are special education teachers eligible to participate?

A

Yes, provided that they have at least ten math students who are expected to take the TCAP math test in the spring. Students taking alternative assessments do not count toward the requirement of ten.

Q

What happens if I meet the minimum requirement (10 students) now, but over the course of the year some of my students leave the district or are transferred out of my class, so that I no longer have ten when the TCAP is administered? Will I be dropped from the experiment?

A

Teachers will not be dropped from the experiment, as long as the ten students they have in the fall are expected to take the TCAP in the spring. If students leave a teacher’s class for reasons beyond his/her control, the teacher is permitted to remain in the experiment. However, that teacher would not be permitted to participate the following school year if his/her class enrollment does not go back up above ten at the start of the next fall.

Q

Suppose I sign up to participate this year (2006-07), but in 2007-08 I take a leave of absence for one year. Will I be permitted to rejoin the experiment when I come back from my leave?

A

No. If a teacher leaves the teaching position, even for a leave of absence, we will need to replace him/her. It would not be fair to the replacement participant if we bumped him/her when a teacher returns to his/her position.

Q

If I stop teaching math for one year, will I automatically rejoin the experiment when I return?

A

No, when a teacher stops teaching math, we will need to replace him/her. That teacher will not automatically rejoin the experiment, even if he/she starts to teach math again.

Q

Are teachers competing against one another for bonuses

A

No. Teachers are judged against a standard based on past performance of Nashville teachers. This standard was determined at the beginning of the experiment and will remain fixed throughout the duration of the three-year experiment. All teachers have the opportunity to improve their performance. In principle, all teachers can end up exceeding this determined standard and earn a bonus.

Q

Is NCPI using the TVAAS (Tennessee Value Added Assessment System) to determine which teachers receive bonuses?

A

We are not using TVAAS. Our procedure for determining which teachers receive bonuses is described below in the question, “How are bonuses determined?”

Q

How are bonuses determined?

A

Bonuses are based on two factors: (1) the progress of a teacher’s math students over the school year (as measured by their gains on TCAP), and (2) the progress of teacher’s non-math students over the school year (also measured by gains on TCAP).

It is easiest to explain this procedure using a hypothetical table. Suppose the table below (Figure 1) represents a roster of math students for Mr. Brown. All teachers assigned to the treatment group will receive a table like this detailing test results for each of their students, once the test results have been made available by the state and district. This roster includes each student’s TCAP score from spring 2006 (column 2) and spring 2007 (column 4). Between these two columns is another column containing benchmarks based on 2007 scores for all students in Tennessee at this grade level.

Figure 1: Teacher-Mr. Brown, Mathematics

Student	Indiv. 2006 Math TCAP Score	State Math Benchmarks for 2007	Indiv. 2007 Math TCAP Score	Indiv. Difference from State Benchmark
J. Smith	250	270	285	+15
M.  King	260	279	277	-2
F. Esposito	265	284	302	+18
L. Davis	255	273	267	-6
A. Aziz	230	255	258	+3
J. Ruiz	242	263	263	0
A. Johnson	254	274	288	+14
E. Jones	261	280	297	+17
W. Graham	248	269	275	+6
T.  Sawyer	237	260	271	+11
P. Morel	244	265	262	-3
V. Fleming	251	270	285	+15
I. Petrovitch	269	282	285	+3
L. Belkin	253	273	280	+7
Class Average Difference		+7

For example, the first student on the roster is John Smith. John scored 250 on the TCAP in spring of 2006. The state benchmark score for John is 270. This represents the statewide average score in 2007 of students who, like John, had a score of 250 in 2006. In column 4, we see that John Smith’s own 2007 score is 285. Thus, John gained 15 points more than the average student in Tennessee who started at the same level. We record this as a +15 in column 5.

We do likewise for the other students on the roster. Each student’s own gain is compared to the benchmark gain of similar students in the state, and the differences, plus or minus, are recorded in the final column.

At the bottom of column 5, we have averaged the differences for all students. Mr. Brown’s students gained 7 points more, on average, than similar students statewide. To find out whether Mr. Brown has earned a bonus, we compare his score of +7 to targets based on the performance of Nashville mathematics teachers in recent years. The lowest target is based on the top 20% of Nashville teachers from 2004 to 2006. If Mr. Brown’s +7 is equal to their performance, he will qualify for the lowest-level bonus, $5000. To earn a $10,000 bonus, Mr. Brown would need to exceed the performance of 85% of Nashville teachers from 2004 to 2006. And to qualify for a $15,000 bonus, he would need to exceed the performance of 95% of Nashville teachers over that time period.

Because these targets are based on historical performance in the district, they stay fixed throughout the experiment. That means it is possible for all teachers in the treatment group to meet these targets and earn bonuses. They are not competing with each other. Rather, teachers are competing with historical targets that do not change.

If Mr. Brown teaches subjects other than math, a table similar to Figure 1 will be completed for each of those subjects that is assessed by TCAP (English/language arts, science, and social studies, in addition to math). To receive his full math bonus, the students Mr. Brown teaches in other subjects must perform at an acceptable level. This is defined as the district average gain (again, that is based on historical targets).

Suppose, to continue this example, that Mr. Brown has 14 students in math and 14 students in science. If Mr. Brown qualified for a $10,000 bonus in math, and the average gain of his 14 science students equals the district’s historical average gain, relative to state benchmarks, he will receive the full $10,000. If his science students do not make this district average gain, his bonus will be reduced by the share of his students who take science. Since 50% of his students are in math and 50% are in science, he would lose 50% of his bonus and receive $5000 rather than $10,000.

Students in subjects that are not tested under TCAP (such as music, art, and foreign languages) do not affect your bonus.

Q

Some teachers might try to improve their scores by encouraging certain students to stay home on test day or by coaching them during the exam. Do you have any way to deter this?

A

News reports from other parts of the country indicate this can be a problem. As a result, we have developed the following procedures to safeguard the integrity of the experiment.

First, we examine attendance during TCAP. If treatment and control groups in the same school have essentially the same attendance rate, we will conclude that no suspicious behavior occurred. Even if attendance in the treatment group is lower, it will be deemed acceptable for purposes of this experiment as long as it exceeds the requirements of No Child Left Behind. If neither of these conditions is satisfied, we will conduct further analysis. Treatment teachers with the worst attendance may be subject to one of the following penalties: (1) a reduction in their bonus (if they earned one) or (2) being dropped from the experiment the following year. We emphasize that penalties will be applied only in the worst cases. If the attendance of a treatment teacher is not significantly worse than the rest of his/her school, that teacher will not face any penalties.

Q

When will teachers receive their bonuses?

A

NCPI must wait until the district has received test results before it can calculate who has earned a bonus. We must also wait for the state to furnish us the data needed to compute state benchmarks. All of this should occur before the beginning of subsequent school years, but there could be delays. Once NCPI has the data it requires, we will inform the district which teachers have earned bonuses. If all goes well, teachers should be notified whether they have earned a bonus by the beginning of the subsequent school year and receive the bonus shortly thereafter. If there are questions about the validity of students’ scores, payment of bonuses may e delayed until further investigation is completed.

Q

Do students who are not in my class for the full year count toward my bonus?

A

We follow the rules established under No Child Left Behind. To count for purposes of this experiment, a student must be continuously enrolled in a teacher’s class, starting no later than the 20th day of the school year, until the time of test administration.

Thus, a student who leaves a class half-way through the year would not affect a teacher’s bonus. However, this student still counts toward determining a teacher’s eligibility for participation, because eligibility is based on enrollment at the time participants were chosen in the fall of 2006.

Q

Do students who have missing test scores count toward my bonus?

A

Students with missing test scores from the previous school year do not count, because we cannot calculate their gains. However, we are working with MNPS administrators to obtain scores for as many students as possible, including students who were not enrolled in MNPS last year but who took TCAP elsewhere in the state.

Q

How may I obtain a copy of the bonus calculation guide for teachers?

A

Click here for the Guide to Calculating Monetary Bonuses for Teachers.

Q

Why is NCPI conducting this experiment?

A

Incentives for teachers are being widely discussed. They have been enacted by some state legislatures (for example, Texas and Florida) and are under consideration elsewhere. However, very little is known about the effectiveness of these incentives. We believe that more should be known about their consequences before legislatures and other public officials rush to put such policies in place.

Q

Why are only middle school math teachers eligible?

A

First, there were not sufficient funds to set up an experiment that would cover teachers of all subjects at all grade levels. Second, previous research with achievement test data has shown that the effects of math teachers can be identified more readily than the effects of teachers in other subjects.

We chose middle school rather than elementary school because middle school math teachers, on average, work with larger number of students than do elementary math teachers. Having a large number of students improves the quality of the data obtained from the experiment.

We chose middle school rather than high school because middle school students in Tennessee take the TCAP, a vertically-linked exam on which it is possible to measure gains from year to year. At higher grade levels, students take a variety of exams that lack this property.

Q

What will the researchers do with the results of this experiment?

A

NCPI researchers will write a report describing the effect that incentives had on mathematics achievement. Regardless of whether the effect was positive, negative, or zero, we will report the findings. The conclusion will be based on a comparison of treatment to control group teachers. Only group comparisons will be used, not comparisons of individuals. No information that could identify individuals will be reported by researchers.

We will also write a report describing the other effects of the incentive plan on schools and teachers, as revealed by the surveys that participants will fill out.

These reports will be accessible through the NCPI website, NCPI Publications.