Planning the Project Design

Using Pat Hutchings’s taxonomy of SoTL questions can make it easier to think about project design, particularly the most common kinds: projects that ask “what is?” or “what works?”

“What Is?” Designs

“What is?” projects are thick descriptions, snapshots of learning dynamics, or other detailed analyses of what student learning looks like, or what’s happening during the moments of learning.  The size and scope can range from small (e.g., a case study of one student, a thin slice of learning in a single class period) to large (many students [a “large n”], multiple activities occurring over time).

“What Works?” Designs

“What works?” projects assess the effectiveness of or change caused by an activity, tool, or other intervention.  According to the scientific method, a traditional experiment would have two randomly assigned groups: one who receives the intervention and one serving as a control.  In SoTL, however, this isn’t always possible.  It can be challenging to access two sections of a single course, students are not randomly assigned to courses, and there are so many variables affecting the students’ learning that it’s difficult to design a “clean” experiment.

Some SoTL theorists have pointed out that it may not even be particularly desirable to aim for such “cleanness” in design. For instance, Catherine Lewis, Rebecca Perry, and Aki Murata explain in “How Should Research Contribute to Instructional Improvement?” (2006) that

“The very qualities that suit an innovation to controlled trial may handicap it at the later stage of broad dissemination” by … emphasizing “fidelity to the original design that stifles continuing improvement”; whereas “the very characteristics that make an innovation…difficult to study in a well-controlled fashion” make it more applicable and usable by others. Some scholars are drawn to the “general proof” of controlled experimental design and fail to recognize the “local proof” of local and seemingly messy findings that result from less controlled research…. “In part, this is because we often choose the wrong endpoint for research: causal proof of an innovation’s effectiveness under controlled circumstances at initial sites, rather than ‘legs’ and effectiveness at subsequent sites of spread…. In short, it should be incumbent on researchers to justify their choice of research methods based on the endpoint—instructional improvement at sites of spread [dissemination and broader use]—not just the premature stopping-point, proof of innovation effectiveness under controlled conditions. The seeds of dissemination success or failure may well be planted at the outset of innovation.”   (emphasis in original; p. 8 )

Similarly, Donald Schon describes a “dilemma of rigor or relevance”:

“In the very topography of professional practice, there is a high, hard ground overlooking a swamp. On the high ground, manageable problems lend themselves to solution through the use of research-based theory and technique. In the swampy lowlands, problems are messy and confusing and incapable of technical solution. The irony of the situation is that the problems of the high ground tend to be relatively unimportant to individuals or to the society at large, however great their technical interest may be, while in the swamp lie the greatest problems of human concern. The practitioner is confronted with a choice. Shall he remain on the high ground where he can solve relatively unimportant problems according to his standards of rigor, or shall he descend to the swamp of important problems where he cannot be rigorous in any way he knows how to describe?” (p. 28)

As a result, SoTL researchers attempting comparisons try to use groups that are as similar as possible to measure the effect of an activity as precisely as possible. If the groups are similar, the differences in outcomes are more likely to have arisen from the intervention, rather than from innate differences across groups, but the researchers are always explicit about the specific groups, context, and potential limitations.

The video below reveals some explorations of the major controversies, debates, and tensions in SoTL.

For guidance on this stage of a SoTL project, see
“Project Design: Structuring the Learning Activities and Data Collection.”

Next →