January 07, 2014

What am I trying to encourage with my exam policies?

I'm trying to figure how to handle giving quizzes and exam next semester in my algebra-based physics courses, my intro astronomy course and my general education course in musical acoustics.

There has been much talk online recently about Standards Based Grading (SBG) and related assessment strategies. I'm not diving fully into the SBG waters, and currently my issue isn't directly related to going towards SBG.  The reason I mention SBG is to give some context.

Introductory Physics Courses

A few years ago when I learned about SBG, I sort of had the wrong idea of how it was supposed to be implemented. I liked the philosophy which allowed for students to learn at their own pace and to be reassessed on understanding of the standards.  I also liked the idea of using student-made screencasts (Thanks to Andy Rundquist for leading me down this path) as assessment methods.  Because I get to hear the students explain the physics in their own words, I can really find out what they understand and what they are simply regurgitating from class or the book.

Grades in intro physics are made up of the following parts: online reflections of what they did in class and read in the textbook, screencasts done for homework, lab reports, weekly quizzes, midterm exams and a final. I consider the lab reports to be drafts which can be corrected and submitted until they are satisfactory. I also consider the screencast homework assignments to be practice for taking quizzes and exams, so I provide feedback on the screencasts and allow them to be resubmitted as many times as needed until correct.

Quizzes and exams are done in a traditional way - all the students spread out as far away from each other in the classroom and work independently on the quiz or exam for a set amount of time. For quizzes, I provide relevant (and sometimes not-so-relevant) equations, but for the exams students prepare their own equation sheet. I usually give 20 minutes for a quiz and 2 hours for an exam.

General education courses - Intro Astronomy and Physics of Sound, Music and Hearing

In the gen ed courses I do not use screencasts. The only homework that the students are required to do is the classroom reflections. Astronomy is not a lab course, so there are no lab reports, but they do have to do a semester-long astronomy journal project. In the acoustics class, students design and build their own musical instrument.  I'm pretty happy with those parts of the grading process.

But the exams are something else. Again, I have typically given "traditional" type exams where all students work independently. I typically supply equations for these classes.

There is a pattern that is starting to emerge over the last few semesters in astronomy. The first part of the pattern is that on the first exam the class average is somewhere in the mid-60% to mid-70% range. For many students it is shockingly low. However, in the 10-ish years I've been teaching the class, the average on the first exam has never strayed far from this mark.  Typically we have a discussion of how now they know how the exam will be structured (even though we discussed it thoroughly beforehand) and that they should think carefully about what changes they need to make in preparing for the next exam. I've also been weighting the first exam less than later exams in recent years to try to alleviate concern that their grade is sunk after one poor exam. The next part of the pattern is that on the second exam (out of three midterm exams) the class average goes down. Significantly down. In most semesters before the last 3, the class average would rise to right below about 80%.  More recently, the average has declined to the low 60% range. 

Frustrated by this pattern, I offered to allow group exams in astronomy on the third midterm.  Working together, the students were able to significantly bring up their scores, although implementing the group exam brings in its own set of challenges in terms of how I score it fairly.

What am I really trying to encourage?

There are some maxims that are sort of swirling around in my head whenever I think about what I'm going to do next semester. One is the saying about how students don't really respond to what you want them to do (or what's best for them) but they will respond to what they are graded on. I guess I can't really think of the exact saying right now, but I think a lot about how to incentivize the intrinsic motivation to pursue deep learning without having to provide the extrinsic motivation of points towards a grade.

The other related thought that I can't quite decide how to address is the idea that if I want to encourage a type of behavior or thinking, then it SHOULD be a part of the grade somehow.

So for example, last semester in astronomy we used the lecture tutorials by the CAPER team as purely formative assessments. Students were told they would not be graded on them, so they should work together and feel free to make mistakes on them that we would correct in class.  My class never fully bought into the idea taking the tutorials seriously as a way of being actively engaged in the class. Even after the first exam had 80% of the questions based directly on the lecture tutorials, and the students themselves recognized how much of the exam was based on the tutorials they did not believe that collaborating with others on the tutorials was necessary.

And why should they have? I was not going to be rewarding them for working with others as a part of their grade, after all. I think that perhaps if group exams were a part of the course from the start, they would have reason to work with others in the class from the beginning.

But, what about the general physics course?  I believe Eric Mazur's Harvard course has some form of open-book, open-note policy on quizzes and exams. Others have used group exams in these courses.  What am I trying to encourage?  I think I am trying to encourage students to work together collaboratively, but am I grading that way?  Should I be? Isn't part of the course figuring out how to take quizzes and exams by yourself?

The real reason I need to figure this out

I have a conference that is going to take me away from school the last week of the semester before finals. I am not happy with this schedule, but there is not much I can do about it right now. What I'd like to do if possible is eliminate in-class exams. Since I typically give three mid-term exams, that effectively gives me back all my time I would be missing at the end of the semester…although it's really never the same. But if I give take-home exams, for example, how should they be structured? Do I explicitly forbid collaboration and trust the students? That seems to go against the classroom dynamic that I would like to foster of students working together. Do I explicitly encourage students to group up and work on it?  That would seem to disadvantage students who have busy work and home schedules and cannot easily pop back and forth to campus.

The one idea I've had is to give the exams as take-home exams and allow for students to group up if they want. But instead of them handing in the exam, have them make screencasts for each problem on the exam. That way, I hear each student explain it in their own words, just like the homework. I just don't know if I can grade that many screencasts in a reasonable amount of time.

TL;DR

How can I improve the way I assess and evaluate students next term? How closely does the grading policy align with my philosophy on learning and what can I do to improve that?

5 comments:

Pat said...

http://psych.wustl.edu/memory/Roddy%20article%20PDF%27s/Roediger%20&%20Karpicke%20(2006)_PsychSci.pdf

I went from teaching a daily Statics class to a block twice a week. I thought the level of preparedness was going to go down drastically so I thought I better quiz them everyday.
I posted all lectures online and quizzed students on Monday at the start of class over the lectures we would cover for the week. The lectures had solved example problems, multiple choice review questions, and basic info. I cut and pasted directly from the lectures. No surprises. They could use any hand written notes on the quiz. This quiz is individual.
On Wednesday, we would cover the last of the material and because students had already read and studied the lectures, in class time was more in depth discussions and a more difficult problems. We would work a difficult problem in class. Then they would get a group quiz that covered the material and they had open book open notes. But challenging quizzes.
This was the best prepared group of students that I have ever had. They destroyed my tests. Questions that would have had my past students fumbling, 90% of the class were solving them in "none traditional" methods. They owned their learning.
Half way through the quarter in our faculty journal club, a colleague shared the paper I shared above and what I was experiencing in the classroom was what is shared in this paper.
I have all students in teams of four and they have team homework and I only grade one quiz for the Wednesday quiz - the one they select.
Tests are still individual and the final.
I also had students right down what they were still struggling with on the back of the Monday quiz - this was amazing because the were reaching meta-cognition faster. I would then tailor the Wed class for their struggles and then push them all.

First test I pulled most of questions from homework, examples, and material that they should have seen. They did quite well on the test. But I think this encouraged them to master the material for the rest of the quarter. I could have made a challenging test to start - but I just kept on increasing the difficulty instead. But as I said earlier - they where prepared. They owned the class.

I have many of them in the second class, Mechanics of Materials, and I'm doing the same thing - quizzes everyday. All individual this time. I'll let you know how it goes.

Pat said...
This comment has been removed by a blog administrator.
Andrew said...

Thanks for sharing this article and your experience with your class, Pat!

I'm really impressed with the statement: "They owned their learning." That's got to be an incredible experience.

You've given me a lot to think about. Thanks for the feedback!

Trevor said...

I'm currently researching and developing how to incorporate SBG into my college-prep physics class next school year, and I found your thoughts on midterms and exams englightening. I'm having issues myself figuring out how to implement them and keep with the SBG philosophy.

You mentioned student screencasting as homework. Do you have a post here where you go more into depth on how you do that? Policies, the technology you use, etc.? I think having students use screencasting as a way of reassessment is a fantastic idea, and I'll definitely be looking more into it.

Andrew said...

Trevor, thanks for your comment. Here's a post I did on screencasts, but I should update that or more fully explain it.

Good luck with implementing SBG! I hope to hear more about it from you.