The mixed methods research design involves both primary data collection and analysis of primary and secondary data in two school districts, Milwaukee Public Schools and Dallas Independent School District. We measure student access to digital tools, where a student has the opportunity to use digital tools in a classroom or other educational setting, and actual use (or enactment) of digital tools in the educational setting, including measures of intensity, duration and quality.
We developed a standardized Classroom Observation Instrument to evaluate the nature of digital tools themselves and their implementation, as well as the quality of learning opportunities in digital and blended instructional settings within the instructional core and the elements that contribute to quality. The instrument is based on existing research on the integration of technological tools into classrooms, as well as indicators of quality elements in the instructional core in general and for digital instruction in particular.
A second instrument of data collection that we developed is a Digital Tool Walk Instrument designed for a quicker, in-person overview of the types of digital tools in use and how they are being used in the educational setting. This instrument collects information on the functionality of digital tools and instructional strategies and practices observed at the time of the “walk through.” We recommend its use for a wider, surface scrape of practices and implementation to help districts and schools map overall patterns in the integration of digital tools.
Recognizing that with many digital learning tools, online content is often experienced only by students, we developed a third instrument, Curricular Relevance and Responsivity Protocol, for assessing the level of cultural relevance and responsiveness in online or digital courses. Educators could also use this tool to do shorter digital “walk throughs” of course content, for example, to scan for promising or problematic content.
Also for use in evaluating online content, we developed an Authentic Online Work Rubric that enables users to evaluate the extent to which online courses provide opportunities for higher-order thinking and real-world relevance, two primary components of authentic work identified in research. The higher-order thinking scale component of the rubric is designed to measure the extent to which students are asked to think deeply and critically about course content and to generate new knowledge. The real-world relevance scale aims to identify the extent to which course content resonates with or is applicable to students’ lives, interests, and/or aspirations.
We are also conducting structured interviews with district and school-level administrators and support staff to characterize and understand how malleable factors such as organizational capacity and inter-organizational coordination; staffing, training and support decisions, and policy guidance and requirements for implementing the technology initiatives in school districts influences access to and the effectiveness of digital tools in improving student outcomes and reducing achievement gaps.
We are employing econometric methods in analyzing school record data on students in classrooms where the digital tools are being accessed and used, which include student characteristics and educational outcomes. In Milwaukee Public Schools, we are also linking student data with administrative data from contracted providers of the digital tools, which includes detailed (session-level) information on the intensity of student use, active and idle time, course-taking and performance, as well as other measures of digital tool enactment in and outside of the classroom. We draw on these measures in quantitatively analyzing the relationship between the use of digital tools and student learning and achievement and the malleable factors that contribute to their effectiveness (or limitations) in improving student outcomes and reducing achievement gaps. In addition, we have linked National Student Clearinghouse data and Unemployment Insurance data that allow us to evaluate student outcomes after they leave the K-12 education system.