Rubrics Matter: Our Upcoming Peer Assessment

by Derek Bruff

At yesterday’s study group meeting, we briefly discussed the upcoming peer assessment activity in #MapMOOC. I’m excited to try and tell a story through a map, but I’m a bit worried about the quality of feedback I’ll receive from other MOOC students. Given the research on peer assessment conducted by the CFT’s graduate assistant Katie McEwen last year, I’m convinced that the effectiveness of a peer assessment activity depends largely on the quality of the rubric. And the rubric in #MapMOOC is unfortunately fairly simple:

  • Does this map tell a complete story?
  • Is the story that this map tells presented in a compelling way?
  • Is the map designed in a way that reflects the use of best practices in cartographic design and geospatial analysis?
  • Does this map have an aesthetic look and feel that reinforces the objectives of the story it tells?

We’re asked to give our peers a 0, 1, 2, or 3 on each of these questions. What do each of those scores mean in the context of these four questions? Who knows. I might give a 1 to something you feel deserves a 3. Without clearer criteria, there’s not likely to be any consistency in how the maps are assessed across the course.

This is, in fact, I point I made at a workshop last week for our TA Orientation staff. After giving the workshop participants some student work on a math problem, I asked them to assign a grade to that work between 0 and 5. One of the student responses received a 0 from one participant and a 5 from another! Without a clear, somewhat objective rubric, there’s no consistency in grading nor any real way to interpret a score one receives.

The #MapMOOC peer assessment also includes a free-response question meant to encourage students to give each other specific feedback on their maps. Given the widely varying backgrounds and motivation levels of students in this course, I’m not expecting to get useful feedback from all of my assessors–maybe one if I’m lucky.

With that somewhat depressing context, I’ll reiterate that I’m looking forward to building a map that tells a story. I’m also looking forward to seeing the maps you all build! We might not receive very useful feedback on our work from our fellow MOOC participants, but I’m glad to know that I’ll have a local audience interested in what I make!

Image: “The Calm After the Show,” Thomas Hawk, Flickr (CC)

Posted by on August 13, 2013 in Reflections


Small Group Discussion Meeting

Spectacular!

In a small group of four, we very informally discussed what we thought would be the “ideal MOOC.”  We were so engaged in the conversation that we actually didn’t even know that more than 90 minutes had passed and had to “force-end” it.

Professor Robinson has greatly inspired us.  Many of the things that we discussed were not necessarily perceived weaknesses with his course, but were things that came to our mind as a result of being part of this MOOC.  All in all, we came to the conclusion that this is certainly NOT an easy thing, and if anything feels that just putting together a few videos constitutes the creation of a MOOC, they are very mistaken.

Our outstanding conversation went quickly from one topic to another, but here were some of the things that we discussed:

  1. Consideration of your audience and how you want to “brand yourself;”
  2. Modes of presentation of materials:  text? audio? video?
  3. Instructions for assignments:  I learned today about the difference between the squares and the circles on those quizzes, did you notice them?
  4. Depth of coverage of material:  superficial?  excessively detailed?
  5. Language:  Not all of your participants will be English speakers!  Will they get your jokes and your American cultural references (i.e. Whole Foods, Trader Joe’s?)
  6. And, we also spoke at length about those discussion forums.   Ah, those forums……

We will have a homework session, tomorrow Friday August 9, at 1:00pm.

Posted by on August 8, 2013 in News


Group Meeting: 5 August 2013

We had a very small, but productive meeting today.  We had a very lengthy discussion of two topics:

a)       Gender and MOOCs — Specifically in terms of the reception of the material by the learners.

b)       How to best safeguard against cross-gender/cross-cultural issues in teaching a MOOC.

We’re dealing with large worldwide populations, and it’s not quite evident that many even think about this.  Is American humor appropriate for the MOOC environment?  Is a subgroup of American society’s humor appropriate for the international MOOC environment?

We also helped one group member understand one element of a mapping exercise.

Finally, we’re now at the point to where we can start the “what happens now” discussion.  I’d like to challenge all of you with the coming next week having thought about:

a)       Would you be interested in working on a collaboration in writing up:

i) An informal write up of the experience for internal use at Vanderbilt (i.e. to publish on the Vandy website);

ii)  A more formal article to be sent out to a scholarly journal

b)       What would those projects look like?

I mentioned a “faux ethnographic” study, in which we described each participant’s experience.  “Subject A was a graduate student in Belarus literature and she…..” It’s only an idea.

Todd

Posted by on August 5, 2013 in Meeting Notes, News


Today’s Informal Session

Unfortunately, I am suffering from seasonal allergies. Suffering from seasonal allergies and living in Nashville actually encompass several of the sets that Professor Robinson describes in Lesson 2: equals, overlaps (although I would say that equals is more efficient), contains and covers.

I’d like to ask one of the attendees of today’s afternoon session to take the lead on the discussion. Remember, the goal of this session is not to have an organized discussion, but to just sit around and “chit-chat.” But, you certainly can spend your time however you want. I will be in the office until 2:45, then I will leave to go home to take my Day-Quil. Thank you for your help!
Thank you, theairquality.com, for this lovely image.

Posted by on August 1, 2013 in News, ,


Second Week Reflections: Social Learning in a MOOC

Our discussion on Monday of the social aspects of learning in a MOOC was an interesting one. It’s clear that many of us are benefiting from the interactions we’re having with our local study group, although I wonder what our study group members who weren’t present on Monday would say about that. Perhaps some of them are getting along just fine in the MOOC without interacting with local colleagues…

For those who do benefit from the local group, I posited on Monday that it was the smallness of the group that mattered more than its localness. It’s certainly challenging to build relationships with other MOOC students through the Coursera discussion boards–there are thousands of students, the boards are chaotic, and there are limited social tools for connecting with others. I can imagine building a small learning community in a virtual setting–I do it all the time on Twitter–but I don’t see the Coursera MOOC environment as particularly conducive to that.

But perhaps, as Todd indicated, it’s the persistence of the relationships in a learning community that is more important than its size.  I knew many of you before this MOOC began, and I expect to interact with many of you regularly after the MOOC is over. Those persistent relationships are helping motivate me to keep up with the course and see what I can learn from it–so that I can participate in our study group conversations.

There’s a motivational piece to social learning, but there’s also a cognitive one. We know from the research on how people learn that, as we learn something, we need feedback on our learning. We need to be able to try out ideas and techniques and solutions, to see what works and makes sense and clarifies. In some contexts, that can be done without others (e.g. checking your answer in the back of your math textbook), but in most contexts, we need to receive this feedback from others. Expert feedback is usually best, but feedback from other learners can be very helpful, too. (Jargon alert: We call this kind of feedback “formative assessment.”)

Dani’s unfortunate experience with peer assessment in another MOOC points to the challenges of matching students at random to provide feedback to each other. Better, I think, to provide feedback within a “local” (small, persistent) learning community, where students know and, to some extent, trust each other. For students taking a MOOC who don’t have a study group like ours, where do they get this kind of feedback? The discussion forums? The forums could serve this purpose, if well structured by the instructor, but perhaps only for some students and in some teaching contexts. As I mentioned on Monday, I hope to see some technical innovations in MOOCspace that help address this challenge.

Thanks for another stimulating discussion this week! I should add that what’s not reflected in the whiteboard notes above is our discussion of the roles that MOOCs might play in the higher education ecosystem. There’s also no geography in the notes! I expect that we’ll spend more time next week actually discussing the course content.

Posted by on July 30, 2013 in Meeting Notes, Reflections


Group Meeting: 29 July 2013

Image is royalty-free and in public domain from photos.com.I’m certain that Derek Bruff will have something to post, but I’d like to make a few thoughts known, in relation to today’s conversation.

First and foremost, I feel that my learning is facilitated by contact with others.  I’ve come to that conclusion.  I can successfully complete assignments and process material on my own, but the actual learning takes place when I am able to collaborate and synthesize with others.  It’s a surprising revelation for me.  Perhaps I’m not as introverted as I think that I am?  No.  I’m pretty introverted.

We talked a lot today about what collaboration with others in the MOOC environment would look like.  Some felt as though it can be achieved 100% through technology, while others went to the other extreme.  I’m somewhere in the middle.  But, I do feel that we were all in agreement in that there needed to be persistent relationships in the process; otherwise it does not seem like learning.

Another point that resonated with me today is whether or not the MOOC environment is best suited to present opportunities in learning for college credit; or for professional development; or perhaps something in-betweenAll threeNone?   The one thing that I can say is that this experience has convinced me that language learning through this type of medium is not appropriate.  I am one for the blended model.

I, myself, still feel a sense of cynicism towards this class.  I feel as though we are basically going through a set of motions towards an undefined goal.  Will this goal ever be revealed to us?  Are we learning?  Are we training ourselves?  Are we familiarizing ourselves?  Are we being subjected to a ploy on behalf of Coursera and PSU?

<<Time for a tangent, with the requisite anecdotal revelation about myself.>>  I will now reveal myself as a fierce online student.  I am taking a certificate programme (Canadian) through St. Michael’s College at the University of Toronto, given in conjunction with the Canadian National Institute for Genealogical Studies.  Yes, I will be certified as a professional genealogist!  I must admit that the amount of enthusiasm that i have towards my GIS MOOC is minimal, in comparison to my genealogical studies.  I’d also say that my interest level in both topics is similar.  I could almost posit that my interest level in GIS is stronger, as it is work related.  Read:  paycheck.

I hate to say it, but my response to the big question about “why should I take a MOOC-based course”  is still all of the above (professional development, intellectual inquiry, get some college credit) and none of the above.  I still really don’t know which one is my best match for the GIS MOOC.  But, alas, this answer would be wrong on my exam and force me to have to take it again– because you have to score 100%!

Posted by on July 29, 2013 in Meeting Notes, Reflections


Collaborative Homework Sessions

Today we had our first collaborative work session.  We watched the videos for Lesson 2 and completed the mapping assignment.

I have a few observations.

1.  I really appreciated the ability to “talk things out” with Cliff and Mona today.  I didn’t realize how this mattered in my comprehension of the material in Lesson One versus Lesson Two.  Today, after watching each video, we asked a few pointed questions, such as “What is he trying to achieve?” or “What was that segment about?”  Our discussions demonstrated that we all heard varying amounts of the details.  I realize that I would have missed a lot of the content, without my friends there.  I also appreciated the ability to “synthesize” the content through conversations with others.

2.  Completing the mapping assignment was a whole lot of fun in the group environment.  We helped each other through the maze of ArcGIS menus and explained different concepts,such as layer transparencies, to each other.  This environment kept me moving and motivated to do the entire assignment.

3.  I did much better on the quiz, this time.

4.  The bottom fell out once they left and I felt the same ole “lack of interest or “I don’t care” attitude once I went into the discussion forums.  If I were an undergrad, perhaps I would have posed the famous question “What do I have to do to get an A?”   I realize that perhaps it’s not the discussion forum topics or even the assignment itself, it’s just that it’s very one-way.  I post something and I don’t have synchronous feedback to what I post.  This did not happen in the conversation session.  I was once again motivated to post comments such as “wow, that’s neat” or to fabricate something profound just so I could get my four posts in.  I have the notion that I made much more interesting comments to Cliff and Mona during our discussions than I did in the forums.

What kind of puzzles me about what I have just typed I consider myself a very autonomous learner and I am extremely introverted.  So, what is this thing about me wanting a learning community ?  I often smirk when people use that terminology around me.    Could there be some truth to what they say about the social aspect of learning?

How can we better create this in a MOOC?  Because this is what’s missing for me in this course.

Posted by on July 26, 2013 in Reflections


Finding home and heading out

Quick comment to Derek’s promptings:

1) Great point about an activity that jump starts a MOOC’s involvement. I/we will take that to heart. But I see that opening activity as having two at least two purposes: a) to pull folks into the MOOC, but also b) to help them establish a “posse”, some smaller community within the massiveness of the thing. I am thinking about my college sorority experience (yes, I was a sorority girl :-), another conversation…). Being part of a sorority gave me a home base within a larger institution. I didn’t stick to the sorority of course but the home base was helpful. I think folks need a group identity or predictable interactions (one of the reasons we are thinking about pairs) and then can go out looking for new/provocative sources and ideas. BTW, joining the marching band might have served the same purpose had I played an instrument…

So for me, the question is how do we create an opening exercise that puts folks in a “band” — for teachers that might be a grade, subject matter band or even a band related to the curriculum your district uses ….

Now with respect to “getting out” of one’s own group, I very much like Derek’s proposal for a tool that allows me to follow a handful of those I consider intelligent commentators. I think that creates a new band at a substantive level.

Posted by on July 26, 2013 in Reflections, ,


Informal Small Group Meeting

Today’s Informal Small Group Meeting made me feel like the Maytag Man.

Posted by on July 25, 2013 in Reflections,


First Week Reflections: Getting Started, Getting Lost

A few brief reflections on the first week of the course, now that I’ve completed the Week 1 activities…

The very first course activity, placing oneself on a shared map, generated a surprising amount of discussion, both in the forums and among our study group. I had seen this activity as something of a lark, intended mainly to generate a map that would spark the usual “Ooh, look at all those students around the world!” comments. In another MOOC, that might have been the primary result of this activity, but in this course the activity brought up a number of interesting issues, including the precision of crowdsourced data and privacy issues. As I work with Vanderbilt faculty designing MOOCs, I’ll have to think about similar experiential learning activities that kick start discussions.

Speaking of discussions, the forums in this course are a wonder to behold. As I write this, there are 123 pages of threads. At 25 threads per page, that’s over 3,000 threads… each of which contains one or more posts! By comparison, Vanderbilt’s recently completed “Leading Strategic Innovation in Organizations” MOOC, taught by Professor David Owens, had 185 pages of threads… at the end of the course. And that course had three or four times as many discussion forum posts as Vanderbilt’s other two MOOCs.

The discussions in “Maps and the Geospatial Revolution” aren’t all fluff, either; many are very interesting conversations among enthusiastic students from around the world. (Example: This thread on India-China boundary disputes.) I gather our instructor has been spending a lot of time in the forums. I know I could get lost in there for hours reading and participating in discussions. I appreciate that Dr. Robinson is providing a weekly “digest” of interesting forum threads. Again, I’ll keep that in mind as a recommendation to Vanderbilt MOOC instructors.

I hope that Coursera develops some tools to help students better use the forums in a course like this one. As Cliff mentioned at our meeting on Monday, the Coursera platform handles the volume of posts well. It’s very fast, but I don’t think it’s very clever. One of the challenges of designing an effective MOOC is turning the massive numbers of students from a problem into a strength. I think there’s a long way to go before that challenge is met in the forums.

Here’s one feature that would help: being able to “follow” other students in the course, as one can “follow” others on Twitter. I know I can subscribe to a thread by email, but I’d like to be able to subscribe (in a sense) to other students I find particularly insightful.

These brief reflections are turning not-so-brief, so let me end with a few links to interesting items I’ve collected over the course of the first week.

See you on Monday!

Image: “Choices,” by me, Flickr (CC)

Posted by on July 23, 2013 in Reflections