Procedures

Classroom observations and observer ratings were collected in both Tools and Comparison classrooms. Classrooms were observed three times during the fall and spring semesters of school years 2010-11 (cohort 1) and 2011-12 (cohort 2), with one observation in the fall and two in the spring semester. Two observers visited the classroom to record all classroom activities during the day. One observer completed the Narrative Record and Tools of the Mind fidelity measure (see Fidelity Measures page) as well as an environmental scan of the classroom materials. The second observer completed the Teacher Observation in Preschools (TOP) and the Child Observation in Preschool (COP). At the conclusion of the observation, each observer collaborated to complete the Post Observation Rating Scale (PRS). The classroom observation measures are described below.

Narrative Record

The Narrative Record (Farran & Bilbrey 2004) is an open-ended format for recording descriptive narrative data notes and also rating the activities occurring in the classroom. This system was used in both Tools and Comparison classrooms to determine similarities and differences among them. The focus of the Narrative Record is the whole class; whatever the class as a whole (defined as at least 75% of the children) is doing is coded. The Narrative Record consists of the following items:

Episodes of Time: Each instructional episode is coded for beginning and ending times.  An episode is defined as beginning when there is a change in the method of instruction or a change in the focus of instruction.  Detailed notes are kept about each episode.

Codes for Content of Instruction (Learning Focus) occurring during the episode (e.g., math, reading, code-based language instruction, science, social studies, art, music & movement, and none).

Codes for Level of Instruction provided by the teacher across an episode.  These range from no instruction (0) to highly inferential instruction (4). Inferential means asking open-ended questions; highly inferential involves several turns with inferential questions and follow-up.

Codes for Engagement Level of Students across an episode. These range from very low engagement (1) to extremely and consistently high engagement (5) across the entire episode.

Codes for Type of Activity (Learning Setting) during the episode.

Single setting: Whole Group with or without Teacher (WG), Small Group (SG), Meal, Transition

Multiple settings: Small Groups operating with Centers (SGC), Small Group with Teacher during Centers (SGTC), Out of Room (e.g., outdoors, bathroom, or specials, such as library)

The Narrative Record was adapted to track the following Tools-specific behaviors that could be exhibited in Comparison classrooms as well:

Positive Behavior Reinforcement by the Teacher or Assistant

Behavior Reminders by the Teacher or Assistant

Choral Responses from the Children (children are encouraged to call out answers)

Teacher Paired Activities (meaning the teacher has assigned pairs of children to interact)

Individual Scaffolding by the Teacher or Assistant

Teacher Directed PrivateSpeech (meaning the teacher has directed children to use private speech)

Intentional Teacher Mistakes

Click to view Narrative Record Manual

Click to view screen shot of Narrative Record coding form

Teacher Observation in Preschool (TOP)

The TOP (Bilbrey,Vorhaus, & Farran, 2007) is a system for observing the teacher and assistant’s behaviors in preschool classrooms across a day-long visit when the children are in the classroom. The TOP is based on a series of snapshots of the behaviors of both the teacher and assistant across a period of time when children and teachers are in the room.  Each snapshot may be, by itself, an unreliable piece of information, but collectively, they combine to provide a picture of how the teacher and assistant are spending their time in a classroom. The teacher’s behavior is observed for a 3-second window before scoring. Once scoring has been completed for the teacher, the same procedure is followed for the assistant in the classroom. Teacher and assistant are coded at the beginning of a “sweep;” children are coded immediately afterward. At the end of an observation, a total of 20 sweeps each was collected on the teacher and the assistant. The TOP measures:

How much and to whom the teacher talks and listens

The types of tasks in which the teacher or assistant is engaged (Instruction or Assessment, Management, Administration, Monitoring, and Personal Care, Behavior Approving or Disapproving, Social, None

The level of ongoing instruction or assessment (None, Low, Basic Skills, Some Inferential, and Highly Inferential)

The areas of learning on which the teacher or assistant focuses (Specific Learning Focus: math, literacy, science, social studies,Other Learning Focus: art, music, fine motor, drama, etc., No Learning Focus: no instruction or assessment)

The tone of the interactions teacher or assistant has with the class, vibrant, pleasant, flat or negative

TOP data are not collected when children are out of the room. Only one sweep is allowed during a meal inside the room. The TOP focuses on times when teachers and children could interact in the classroom.

TOP was adapted for use in Tools of the Mind Evaluation.

Click to view TOP Manual (paper coding form included)

Click to view screen shot of TOP on tablet coding form

Child Observation in Preschool (COP)

The COP (Farran et al.2006, 2008 revision) is a system for observing children’s behaviors in preschool classrooms across a day-long visit. The COP is based on a series of snapshots of children’s behaviors across a period of time when children and teachers are in the room. Each snapshot may be, by itself, an unreliable piece of information, but collectively, they combine to provide a picture of how children are spending their time in a classroom (as an aggregate), as well as to provide information about individual differences among children in their activity preferences. A specific child is observed during a 3-second window and then coded across 9 dimensions before the observer moves to the next child. At the end of an observation, a total of 20 sweeps was collected on each child in the classroom.  Consented children are identified by name; all others are identified as “Extra boy” or “Extra girl;” this process ensures that the behaviors of all the children in the classroom are observed. The COP measures:

How much and to whom the children talk and listen

The learning settings in which children are found (Whole Group (with and without teacher) Small Group (with and without teacher), Centers, Transitions, Other: classroom activities not captured by above, such as book reading at the beginning of nap.)

The different types of learning foci of the activities in which children are engaged (eg. Specific Learning Focus: math, literacy, science, social studies Other Learning Focus: art, music, fine motor, drama, etc. No Learning Focus)

The level of involvement of the children: High, Medium High, Medium, Medium Low and Low

As with the TOP, COP codes are only collected when the children are in the room and learning interactions could take place. The COP is not coded during naptime — stopping when the lights go off — or during meal times.  If meals happen in the room, one data collection sweep is allowed.

COP was adapted for use in the Tools of the Mind Evaluation.

Click to view COP Manual (paper coding form included)

Click to view COP screen shot of tablet coding form

POST Observation Rating Scale (PRS)

The PRS (Yun, Farran, Lipsey, Vorhaus, & Meador, 2010) is completed immediately after a classroom is observed and is a 5-point Likert-type researcher-developed scale for rating classroom-level characteristics. This instrument was developed following extensive discussions with the Tools of the Mind curriculum developers, during which they identified classroom attributes that were most likely to be different between Tools classrooms and other early childhood classrooms. The PRS includes items regarding general classroom characteristics as well as teacher practices, classroom activities, and children’s social and academic behaviors.  Both observers completed the PRS independently following the visit and then combined their ratings into a single consensus set of ratings used for analysis. PRS is a supplement meant to be used with a full day of classroom observation.  We do not know if this scale can be used independently of the observation measures.

Click to view PRS Manual