Design and Randomization:

Theory of Change

In order to benefit from preschool and master the literacy, math, and social skills they will need to participate in formal schooling, young children must be able to think, attend, and remember in an intentional and purposeful way. Tools of the Mind is designed to initiate and support the change from reactivity to active and focused thinking. This change is facilitated through specific teacher practices that should help children move from assisted to independent learning. Tools of the Mind is based on an interactive sequence of change whereby teachers use assessment and scaffolding to tailor their use and modeling of specific tactics. These tactics are internalized by their students as cognitive tools, which are then used independently and manifested in observable behaviors in the classroom. These learning-related self-regulatory skills then become the foundation that supports the learning of the preschool “target skills” and the academic content that follows preschool. Below is a graphical representation of the Theory of Change model used to guide the selection of child assessment and classroom observation measures used in this study.

Click to view a pdf of the Theory of Change

Study Design

The research sample from the 2010-11 school year was drawn from five school districts representing a range of urban, rural and suburban locations and serving demographically diverse children in two states (60 classrooms, 32 implementing Tools). A sixth school district provided a 2nd cohort during the 2011-12 school year (20 classrooms, 10 implementing Tools). Experimental condition was randomized at the school level. This scheme helped minimize interactions between experimental and control teachers that might compromise the comparison. The schools were blocked by district, with the largest district divided into two blocks. Within each block, approximately half the schools were assigned to the Tools condition and half to the practice as usual control condition. The teachers in the classrooms assigned to the control condition continued to practice as usual with whatever curriculum they were using and whatever professional development was typical in their setting. The teachers in the Tools condition began the professional development sequence for Tools and implementation of the curriculum during the first year of the study (school year 2009-10 for cohort 1 and school year 2010-11 for cohort 2). The first year was a training and practice year for the teachers and no study data was collected. The research team used this period to develop and practice the classroom observational scheme for collecting fidelity data on the Tools curriculum. Evaluation of the Tools curriculum took place the following school year. Replacement teachers for any of the original teachers that left were trained and their classrooms remained in the study. During the third year of the study (2011-12), training on Tools was offered to each school system in cohort 1 for the control teachers with 3 of the 5 systems choosing to participate in this opportunity. The following year, training on Tools was offered to cohort 2 for the control teachers and this system choose to participate in this opportunity.

Child outcome measures were collected at the beginning and end of the pre-k year, at the end of kindergarten, and at the end of first grade.

Randomization

Random assignment was successful at producing condition groups that did not differ significantly on any demographic variable (ethnicity, gender, IEP status, ELL status, age, teacher education level, years teaching, etc.) as well as entering self regulation and academic achievement scores for cohort 1.

Participants:

Children

During the 2010-11 school year, 828 children (Tools = 477) were seen at the beginning of pre-k and 821 children (Tools = 472) at the end of pre-k. The table below provides demographic information about the participant children from cohort 1.

Note: Cohort 1 participants are from Franklin Special School District, Lebanon Special School District, Wilson County School District, and Cannon County School District in Tennessee as well as Guilford County School System in North Carolina.

During the 2011-12 school year, 267 children (Tools=147) were seen at the beginning of pre-k and 257 children (Tools=142) at the end of pre-k. The table below provides demographic information about the participant children from cohort 2.

Note: Cohort 2 participants are from Alamance-Burlington School System in North Carolina.

Teachers

During the 2010-11 school year, 60 classrooms (Tools = 32) in 45 schools (Tools = 25) participated in the study. The table below provides demographic information about the participating classroom teachers from cohort 1.

During the 2011-12 school year, 20 classrooms (Tools = 10) in 12 schools (Tools = 5) participated in the study. The table below provides demographic information about the participating classroom teachers from cohort 2.

Procedures:

Overall Data Collection

All child assessment and classroom observational data for this project were collected using a paperless tablet data collection system.  This system allowed assessors to record children’s responses on the direct assessment measures while administering the items.  The system had built in checks to notify assessors instantly of missed items, stop rules, ceiling errors, etc.  Similarly, the system for classroom observation measures featured built in access to manuals and allowed observers to navigate quickly to specific portions of the measure depending on what was occurring in the classroom at any given moment.  Assessors/Observers saved their data onto thumb drives and the data was then imported into a main database.  This paperless system eliminated the need for data entry and double entry which allowed for quicker access to the dataset at the conclusion of the assessment and observation cycles.

Click to view more information regarding the Paperless Data Collection System

Direct Assessments

Direct assessments with the participant children were administered in two sessions with each session lasting approximately 25-35 minutes.  The assessments were administered to the participant children individually inside a quiet room within their school.

A team of 16 assessors was utilized to complete the collection of the child level data during the 2010-11 school year, 24 during the 2011-12 school year, 26 during the 2012-13 school year, and 5 during the 2013-14 school year. Most assessment staff were trained to administer one session of the assessment battery with 3 trained in both sessions.  Assessors participated in formal in office training sessions to learn the measures and were required to practice with each other as well as in office staff prior to administering the session to a consented non-study child at a non-study school.  Assessors were required to submit a video of their session with a consented non-study child in order to become certified in the administration of that assessment session.  Videos were scored using a scoring rubric and assessors had to obtain a minimum score of 85 in order to become certified.

Click to view an example of the Assessment Video Scoring Rubric

To measure educational achievement, the Woodcock-Johnson III Tests of Achievement (WJ-III) was used.  To measure self regulatory skills, a battery of self regulation measures was used.

Session 1 (in order as given):

Navy Blue text denotes SR measures, Black text denotes WJ-III subtests

1. Peg Tap
2. Head Toes Knees Shoulders
3. Copy Design
4. Oral Comprehension
5. Applied Problems
6. Quantitative Concepts A & B
7. Picture Vocabulary
8. Passage Comprehension (only administered during the 1st grade followup)

Session 2:

1. Dimensional Change Card Sort
2. Corsi Blocks
3. Letter Word
4. Academic Knowledge A, B, & C
5. Spelling

Classroom Observations

Participating pre-k classrooms were observed three times during the 2010-11 (cohort 1) and 2011-12 (cohort 2) school years. During each classroom observation, two observers were present.  One observer completed the Narrative Record form, Tools of the Mind Fidelity, and Environmental Scan.  The other observer completed the Teacher Observation in Preschool (TOP) and Child Observation in Preschool (COP).  Both observers met together immediately following the observation to complete the Post Observation Rating Scale (PRS).

Classroom observers were trained in one set of the classroom observations (either the COP and TOP or the Narrative Record, Fidelity, and Environmental Scan) by attending in-office training over multiple days. Observers were then required to observe in non-study classrooms with experienced project staff to solidify the material discussed in training and ensure coding reliability.  Once reliable in a non-study classroom, observers were certified to observe in study classrooms.  At the beginning of each of the three observation cycles, reliability was established among all observers in the field.

Cohort 1, 2010-11 Observation Schedule:

Observation 1 Window:

TN (N=30):  10/11/10 through 12/9/10

NC (N=30):  11/8/10 through 12/14/2010

Observation 2 Window:

TN (N=30):   1/18/2011 through 2/18/2011

NC (N=30):  1/19/2011 through 2/25/2011

Observation 3 Window:

TN (N=30):   3/8/2011 through 4/14/2011

NC (N=30):  3/8/2011 through 4/14/2011

Cohort 2, 2011-12 Observation Schedule:

Observation 1 Window:

NC (N=20):  10/25/2011 through 12/2/2011

Observation 2 Window:

NC (N=20):   1/20/2012 through 2/6/2012

Observation 3 Window:

NC (N=20):   3/13/2012 through 3/29/2012

Measures:

Academic Achievement Assessments

Woodcock Johnson III Tests of Achievement (WJ-III):

Woodcock Johnson III Tests of Achievement (Woodcock, McCrew, & Mather, 2001).  The WJ-III is a standardized measure with established reliability and validity and applicability to a wide age range, beginning at age 3. The selected scales for language/literacy focus on emergent literacy and span the areas of decoding letters and words, vocabulary, and comprehension. The selected measures for math/numeracy span number recognition, simple problem solving, and simple math concepts.

WJ subtests used in this study:  Letter Word, Spelling, Picture Vocabulary, Oral Comprehension, Applied Problems, Quantitative Concepts, Academic Knowledge, and *Passage Comprehension (*1st gr only)

Self Regulation Assessments

Peg Tapping:

Peg Tapping (Diamond & Taylor, 1996). In this game, children are asked to tap a peg twice when the experimenter taps once and vice versa. The task requires children to inhibit a natural tendency to mimic the experimenter while remembering the rule for the correct response. Sixteen trials are conducted with 8 one-tap and 8 two-tap trials in random sequence.

Click to view more information regarding the Peg Tapping task

Head Toes Knees Shoulders (HTKS):

Head-to-Toes Task (Cameron et al., in press). In this task, which requires inhibitory control, attention, and working memory (though inhibitory control is the main focus) children are asked to play a game in which they must do the opposite of what the experimenter says. The experimenter instructs children to touch their head (or their toes), but instead of following the command, the children are supposed to do the opposite and touch their toes. The measure is available from Claire Cameron, who gave permission for us to use the measure for this project.

Click to view more information regarding the HTKS task

Copy Design:

Copying Designs Test (Osborn et al., 1984). Children must copy “exactly” a series of simple geometric designs. Error scores are used to assess attention.

Click to view more information regarding the Copy Design task

Dimensional Change Card Sort (DCCS):

Dimensional Change Card Sort (Zelazo, 2006). In this reverse categorization task, children must sort a set of cards based on different sorting criteria given by the examiner. For example, the first sort involves color, the second sort involves shape, and the final sort is a mix of color and shape depending on whether a card has a border or not. An additional level was added during the first grade follow up that involved reversing the rules on the final sort listed above. This task is used to assess attention-shifting.

Click to view more information regarding the DCCS task

Corsi Blocks:

Corsi Blocks Task (Corsi, 1972). The Corsi Blocks task is a visuospatial short-term memory task. Children point to a series of blocks in a pattern both forward and backward. The task has two forward series of patterns and two backward series of patterns. The length of the block pattern increases until the recalled pattern is no longer correct.

Click to view more information regarding the Corsi Block task

Classroom Observations

Narrative Record:

Narrative Record (Bilbrey & Farran, 2004).  The Narrative Record Form is an open-ended format for recording narrative data notes and rating the activities occurring in the classroom as a whole. The Narrative Record is an accounting of the content, activity type, duration, engagement, and instructional level of each segment over the full observational period.

Click to view more information regarding the Narrative Record

Tools of the Mind Curriculum Fidelity:

Tools of the Mind Curriculum Fidelity (Vorhaus & Meador, 2010).  The Tools Fidelity captures the specific Tools curriculum activities that occur within a classroom observation period along with information about the specific implementation steps that occur, and mediators that are used.  In addition, the curriculum developers furnished a list of behaviors that “should not” happen during each activity that are also captured by observers. The Tools Fidelity Measure provides an in-depth look at the degree of curriculum implementation across the year within experimental classrooms.  Although this instrument was used in both Tools and comparison classrooms, relatively few Tools activities were ever coded in comparison rooms.

Click to view more information regarding the Tools of the Mind Fidelity measure

Environmental Scan:

Environmental Scan (Vorhaus, Meador, & Farran, 2010).  The Environmental Scan is an observational tool to gauge a classroom’s environment and materials. It is derived from a list of early childhood materials the Tools of the Mind developers indicate should be available in the classroom. The scan focuses on the play centers and materials accessible to children.

Click to view more information regarding the Environmental Scan

POST Observation Rating Scale (PRS):

Post Observation Rating Scale (Yun, Farran, Lipsey, Vorhaus, & Meador, 2010).  The PRS is completed immediately after a classroom is observed and is a 5-point Likert-type researcher-developed scale for rating classroom-level characteristics. This instrument was developed following extensive discussions with the Tools of the Mind developers during which they identified classroom attributes that were most likely to be different between Tools classrooms and other early childhood classrooms. The PRS includes items regarding general classroom characteristics as well as teacher practices, classroom activities, and children’s social and academic behaviors.  Both observers complete the PRS together following the visit.

Click to view more information regarding the PRS

Teacher Observation in Preschool (TOP):

Teacher Observation in Preschool (Bilbrey,Vorhaus, & Farran, 2007).  The TOP is a system for observing teacher/assistant’s behavior in preschool classrooms and is based on a series of snapshots of the teacher’s and assistant’s behavior across a period of time.  Each snapshot may be by itself an unreliable piece of information but collectively they combine to provide a picture of how the teacher and assistant are spending their time in a classroom. The teacher’s behavior is observed for a 3 second window before scoring. Once scoring has been completed for the teacher, the same procedure is followed for the assistant in the classroom. Teacher and assistant are coded at the beginning of a “sweep” with children coded immediately afterward. At the end of an observation, 20 sweeps have been collected on the teacher and the assistant.

Click to view more information regarding the TOP

Child Observation in Preschool (COP):

Child Observation in Preschool (Farran et al. (2006, 2008 revision). The COP is a system for observing children’s behavior in preschool classrooms and is based on a series of snapshots of children’s behavior across a period of time. Each snapshot may be by itself an unreliable piece of information but collectively they combine to provide a picture of how children are spending their time in a classroom (as an aggregate) as well as information about individual differences among children in their preferences. A specific child is observed during a 3 second window and then coded across 9 dimensions before the observer moves to the next child. At the end of an observation, 20 sweeps were collected on each child in the classroom. Consented children are indentified by name; all others are identified as “Extra boy” or “Extra girl.”

Click to view more information regarding the COP

Child Report Forms (completed by each teacher)

Teachers were asked to complete a child report form on each of their consented children during fall and spring of the prekindergarten year, spring of the kindergarten year, and spring of the first grade year (cohort 1 only).

Cooper-Farran Behavior Rating Scales (CFBRS):

Cooper-Farran Behavior Rating Scales (Cooper & Farran, 1991).  The Cooper-Farran is composed of 37 items in two subscales. The Interpersonal Skills subscale (IPS) includes 21 items and the Work-Related Skills (WRS) subscale includes 16 items. The IPS subscale measures how well children get along with peers and the teacher. The WRS subscale includes items about independent work, compliance with instructions, and memory for instructions. Items are rated on a 1-7 scale with descriptive phrases to “anchor” points 1, 3, 5, and 7.

Click to view more information regarding the CFBRS

Adaptive Language Inventory (ALI):

Adaptive Language Inventory (Feagans & Farran, 1983; Feagans, Fendt, & Farran, 1995).  The ALI focuses on children’s comprehension and use of language in classroom settings in comparison to their peers and has been used both at the preschool and elementary levels. The measure consists of 18 items that focus on comprehension, production, rephrasing, spontaneity, listening, and fluency. Children are rated on a 1-5 scale.

Click to view more information regarding the ALI

Academic Classroom Behavior Record (ACBR):

Academic Classroom Behavior Record (Farran, Bilbrey & Lipsey, 2003).  The Academic Classroom Behavior Record is composed of 9 items.  This scale includes items about school readiness, liking of school, and behavior problems.  The first six items are rated on a 1-7 scale with descriptive phrases to “anchor” points 1,3,5,7.  Items 7 and 9 are multiple choice items.  Item 8 is an item in which the rater checks all answers that apply. This instrument was only used with participating kindergarten and first grade teachers.

Click to view more information regarding the ACBR