Regression Discontinuity Design (RDD)
The age-cutoff regression discontinuity design (RDD) followed two cohorts of children in a representative sample of 155 classrooms across the state of Tennessee. Originally 160 classrooms were selected (40 in each region), but numbers changed slightly because of changes in selected classrooms from the 1st year of participation to the 2nd year. Five classrooms were omitted for the following reasons: the teacher moved to a non-participating school (n = 1); the classroom served a very different population (n = 2); the classroom was no longer a pre-k classroom (n = 1); and the classroom had 3-year-olds rather than 4-year-olds (n = 1).
Individual assessments on the same literacy, language, and math measures used in the RCT were administered at the beginning of kindergarten for the cohort of children who were enrolled in TN‐VPK classrooms the previous school year (treatment condition). The scores for those children were compared with scores obtained from the cohort of children just entering TN‐VPK in those same schools that same school year (control condition), with the proviso that scores were used only from children who then completed the pre-k year. With the difference in age accounted for statistically, this comparison provides another set of estimates of the effects of TN-VPK. These data are fully reported in a technical report titled The Estimated Impact of the Tennessee Voluntary Prekindergarten Program on Children’s Literacy, Language, and Mathematics Skills: Results from a Regression-Discontinuity Design.
The outcome measures used to assess academic achievement in the evaluation of the Tennessee Voluntary Pre-K program (TN-VPK) were selected from the Woodcock-Johnson III (WJ-III) battery of achievement tests. The WJ-III is a nationally normed, individually administered assessment. Two subtests were selected to measure early literacy skills (Letter-Word Identification and Spelling), two were selected to measure early language skills (Picture Vocabulary and Oral Comprehension), and two were selected to measure early math skills (Applied Problems and Quantitative Concepts). These tests are widely used in research to assess the cognitive and academic skills of young children and as outcome measures for early childhood programs. The state of Tennessee was divided into four regions; 40 classrooms were selected from each region to be representative of the state’s program. Each year, in a given region, trained staff assessed students in the treatment condition of the RDD at the beginning of their kindergarten school year, and assessed students in the control condition at the beginning of their pre-kindergarten year.
Letter-Word Identification Letter-Word Identification assesses children’s letter and sight word identification ability. Items include identifying and pronouncing letters and words presented to the child. Spelling Spelling assesses children’s prewriting skills, such as drawing lines and tracing, writing letters, and spelling orally presented words. Picture Vocabulary Picture Vocabulary assesses children’s ability to name objects presented in pictures; it measures early language development and lexical knowledge. Oral Comprehension Oral Comprehension assesses children’s ability to comprehend a short passage and supply the missing word using syntactic and semantic clues; it measures their listening ability and understanding. Applied Problems Applied Problems assesses children’s ability to analyze and solve numerical and spatial problems presented verbally with accompanying pictures of objects. Quantitative Concepts Quantitative Concepts assesses children’s knowledge of mathematical concepts, symbols, and vocabulary, including numbers, shapes, and sequences; it measures aspects of quantitative math knowledge and recognition of patterns in a series of numbers For more information on these tests and the WJ-III achievement battery see: Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock- Johnson III Tests of Achievement. Rolling Meadows, IL: Riverside Publishing
As part of the RDD, trained and reliable members of the PRI research team conducted in-depth systematic observations of the 155 pre-k classrooms, chosen to represent the statewide program. These observations took place during the school year when students in the treatment condition were enrolled in pre-k, the year prior to assessments. Using measures described below, we collected detailed information about the environments to which children who participate in TN-VPK were exposed. A second technical report, Evaluation of the Tennessee Prekindergarten Program: Programmatic Variation Across the State, provides descriptive data regarding the findings of the classroom observations and interviews with teachers, principals, and pre-k coordinators.
The classrooms chosen across the four regions were stratified to match the distributions of types of programs in the state. The specific strata were provided to us by the administrators in the TNDOE Office of Early Learning (Click here to view the proportions of RDD classrooms in each of the strata). In each region, we sent trained staff to observe the learning environments of the participating TN-VPK classrooms. Observers scheduled a visit on a day that was convenient for the teacher but also representative of a typical classroom day. Observations were not conducted on a day when a special activity was planned, typical school day hours were altered, or any other event was scheduled that would prevent observers from seeing what usually occurred on a regular day in the classroom. The observer typically stayed in the classroom the entire day, making sure to remain as unobtrusive as possible. During the visit, the observer used several instruments to record various things about the classroom environment. Brief descriptions of these instruments are listed below.
Early Childhood Environment Rating Scale – Revised Edition (ECERS-R) The ECERS-R (Harms, Clifford, & Cryer, 2005) is one of the most widely-used measures of global quality in early childcare environments. It is used both nationally and internationally. It consists of 43 items that are rated on a 7-point scale, and the items are grouped into 7 subscales. Like most research studies, this study only uses 6 of the subscales: Space & Furnishings, Personal Care Routines, Language & Reasoning, Activities, Interactions, and Program Structure. The TN-VPK staff requested that programs use the ECERS-R as a self evaluation. Administrators requested that the ECERS-R be included in the observations so that they could examine results of a more systematic administration of the instrument. Early Language & Literacy Classroom Observation The ELLCO (Smith & Dickinson, 2002) is a measure of the general language and literacy environment in an early learning classroom. It consists of item rated on a 5-point scale that are combined into 5 subscales: Classroom Structure, Curriculum, Language Environment, Books and Book Reading, and Print and Early Writing. TN-DOE staff also requested the inclusion of the ELLCO; it was an instrument staff members believed would be useful to the program. Narrative Record Form The Narrative Record Form (Farran & Bilbrey, 2004) is an open-ended format for recording narrative data notes about everything that occurs in the classroom. It consists of the following: The categorical variables for Type of Activity, Content of Instruction, and Level of Instruction, combined with the Duration of Time of each episode provide critical information that can be easily analyzed. This instrument was added to the battery by the PRI staff with the approval from TN-DOE. RDD Interviews Click here to view the RDD Teacher Questionnaire Click here to view the RDD Teacher Survey
Relations between children’s gains and these quality variables will are reported in an upcoming publication.