Contract Grading in Teaching Computer Programming

Laurence L. Leff, Ph. D.

Department of Computer Science, Western Illinois University

Macomb IL 61455

CONTRACT GRADING

Contract Grading for Computer Science Classes

Introduction

Contract grading is an alternative to the conventional grading system. In the conventional grading system, exams and homeworks are each assigned a certain percentage of the grade. A numerical grade, usually between 0 and 100, is assigned to each exam and homework. These are then averaged together in the weights specified in the syllabus. The final letter grade for the course is assigned on the basis of the weighted average. For example, often we use the familiar: 70 < average < 80 gives C, 80 < average < 90 gives a B, etc.

Contract grading works differently. The grade is determined as a Boolean function of the assignments completed. For example, if a student does x, y and z, they get a C. If they do the assignments x, y, z and w, they earn a B. If they complete x, y, z, w, u and v, they would earn an A.

The literature on contract grading describes two main types (Fraser, 1990). In the first, the students "negotiate" individually a contract with the professor. This contract specifies what they will do and the grade they will earn for that work. The second type is the one dealt with in this paper. The professor prespecifies the contract. (I will show later how students can be given various options so they can adapt the course to their needs.)

Reasons For Exploring Alternatives To Conventional Grading

Students do prefer contract grading (Fraser, 1990).

More importantly, grading systems in colleges and universities are empirically bankrupt. There is no meaningful relationship between grade point average and performance on the job(Samson, Graue, Weinsten and Walberg, 1984, Summer). That paper reported that a study-weighted mean of 209 correlations in 35 studies showed that the average correlation was 0.155. Although this was a statistically significant correlation, it means that only 2.4% of the variance in occupational performance criteria could be explained by grades and other academic test scores.

In coop jobs, students would have little total time to benefit from on the job learning. Thus, one would expect a greater correlation here between on the job performance and the students’ grades. In fact, there is no correlation. (Fletcher, 1988)

Contract grading is particularly well-suited for computer science classes. Programmer productivity can vary by as much as a factor of ten between the very best professional programmers and the lowest decile of those who are employed. One can set up a strategy for contract grading where the amount of work is a power function of the grade received. That is a B might represent twice as much work as a C. An A might require twice as much work as that for a B. Adjusting a grading policy to accomplish this goal is much harder to do with a system based upon percentages.

The students in your programming classes who graduate from a B. S. program go off and do very different things. Some might do business programming. Others will work at the bit level doing real time control or device driver design. Others might do scientific programming.

In the strategy section, I will show how to design a contract grading system to allow the students to fine-tune the course to emphasize those parts of the subject matter that they are interested in.

Preparing A Grading Strategy

In preparing a grading strategy for contract grading, you must ensure two things. The minimum requirements for a C or a D should correspond to a student output product acceptable both to the professor and to the department. Specifically, it should cover all aspects of the course necessary for the students to succeed in later courses for which your course is a prerequisite. One also should have an anti-procrastination strategy (see section on Anti-procrastination below).

To put it bluntly, assume that every student in your class was to do the bare minimum they could to earn a C. These students would appear in the next semester’s classes. Perhaps, some of these would have your course as a prerequisite. Would the faculty teaching these course be satisfied with those student’s backgrounds?

Also, the students earning a B or A should be required to do more challenging problems (not just more problems) than those earning a C. In particular, the students who wish the letter grade A should be required to do some problems requiring independent reading of material, particularly computer manuals. In my paper on Landamatics (self-citation), I show how C problems correspond to those problems that can be done with a template. Problems associated with earning a B can be done with some modification of the pattern represented by the template.

I have used two methodologies in organizing assignments and specifying a grading policy. I term these genre-point-based contract grading (GPBCG) and assignment-set-grading (ASCG).

In GPBCG, the professor defines several "genres" of points. Each genre represents one of the educational or performance goals that the professor department has defined for the course. They may represent a content- requirement. E. G., in an assembler course, the department might expect that all students know bit-diddling instructions such as the logical AND, logical OR and bit shifting. They may represent a performance goal--that the student write a certain number of lines of assembler as part of the course effort.

Let us take a very simplified example to illustrate GPBCG.

Assume that in a CS3 Assembler course, there were two topics that the professor felt were important. They were bit-diddling and arrays. The latter concerns accessing arrays or tables of data using the indexed addressing capabilities of the hardware. Also, the professor wants to ensure that students do a certain total amount of work in order to complete the course. There might be some combination of problems that would adequately cover bit-diddling and arrays; however the professor feels that just doing these doesn’t represent sufficient lines of Assembler code written and debugged to pass the course.

The professor would define three genres of points:

He would ask that the students produce 30 work points, 10 bit points and 10 array points to pass. The professor might create six problems with genre points assigned as follows:

Problems 1,3,4, and 5 would be ones that cover arrays. And problems 2, 4, 5 and 6 cover bits. Problems 4 and 5 cover both arrays and bits. And notice that problem 5 would represent an extra-long problem--that’s why it has 30 work-points.

Assume there are two students: Joe loves bits. Jill loves arrays. Joe might choose to do 2, 4, and 6. Notice that he is forced to do at least one problem covering arrays so he exposure to this topic. Yet, he can emphasize the bit instructions that interest him. (Maybe, he anticipates working in an environment using embedded microprocessors where bit instructions might be of relatively higher importance.)

Jill can choose problems that emphasize arrays and avoid the bit instructions--yet the genre system forces her to do at least one problem on bits since she needs 10 bit points to pass the course. (She might choose 1,2 and 3)

Another student who likes to kill lots of birds with one stone could simply do problem five.

In my assembler course, I had seventeen different genres of points. During the semester, the students chose problems from a set of one hundred handed out at the beginning of the semester.

Several genres of points simply represent things that the students must show to complete the course, bits, arrays, operating system I/O, macro, subroutines. Thus, all students in the contract grading system are ensured of having done at least one problem in the areas specified by the guidelines of my department. (Note that in a conventional grading system, a student might pass a course without learning anything about some important part. The grades on the other parts of the course would outweigh the zero on some subject, let’s say bits.) It works out that the genre-based system leads to the students choosing about fifteen problems. Of course, some students do a few more or less depending upon how they choose the problems

Two other genres, hard and manual points are only required for an A.

Problems earning manual points require the student to learn some topic on their own--for example floating point numbers. Problems earning hard points are, as expected, those which require cleverness in implementation or algorithm. In previous assembler courses, they might involve writing an emulator for floating point numbers or a problem involving three-dimensional arrays.

Assignment Set Based Contract Grading (ASCG)

ASCG involves preparing an assignment set containing ten to twenty different problems for each major topic of a course. Each assignment set is graded on a letter basis. The students have specific instructions as to which subparts to do for the grade of C, B and an A on the assignment set.

An example from an assignment set in a CS1 course is shown below. Observe that the requirements for each of the three grades all require ten activities. The assignments A through I are the easiest and most closely follow the template. Assignments J to M are intermediate in difficulty. And lastly, the most difficult assignments are last and are designated with letters from N to Q. Note that the A requirements require more difficult problems--not more problems.

On time

C requirements -- 1 program A to I total 6 activities.

B requirements -- 1 program from J to M, total 6 activities

A requirements -- 1 program from N to Q and two more activities from J to M total 6 activities

Slightly Late

C requirements -- 1 program from A to I total 8 activities

B requirements -- 1 program from J to M , total 8 activities

A requirements -- 1 program from N to Q and two more activities from J to M total 8 activities

Very Late

C requirements -- 2 programs from A to I total 12 activities

B requirements -- 2 programs from J to M, total 12 activities

A requirements -- 2 programs from N to Q, and one more activity from J to M total 12 activities

conventional grading -- Do item K

A student completing the assignments on time might hand in problems A,D, F, J, M, N and earn an A on that assignment set. Note that this corresponds to one problem from the extra hard set, two from the intermediate set and three from the regullar set.

Students in classes using contract grading will report that it creates problems for them if they are "procrastinators" [Fraser, 90]. Often, in conventional grading, a professor will assign penalties for lateness. E. G., 10% will be taken off if an assignment is one day late; 20% will be taken off if an assignment is two days late, etc. This discourages the students from leaving all assignments to the last day of class in the conventionally-organized class structure.

In contract grading, the structure should be set up so that students doing the work on time do less total work to earn a grade. In my classes based on GPBCG, students received double credit for getting work in on time. That is, assignments handed in on time got double the stated number of each genre of points. Often, I would have "sneak previews" where students could come in for a special out-of-class lecture on material that would be covered in the second half of the semester. These lectures would occur in the first six weeks of the class. Students would get a small amount of credit just for attending. However, if they did the homework from that lecture, instead of waiting until the material was covered at the normal time, they would get triple the points stated.

In those classes I ran using ASCG, students would get brownie points for handing assignments in on time and for attending "sneak previews." They would also get brownie points for going to the computer laboratory some time the first week after an assignment was assigned and starting work. The students could replace some assignments with "brownie points." Generally, each of the eight assignment sets would require ten problems. However, if a student was conscientious about handing materials in on time, starting early, etc., they would only have to do four of the problems. (The departmental guidelines specifies that the students do a total of eight problems in the semester. Thus, even if a student were to take advantage of all possible bonuses, their total work output would still be above departmental guidelines.)

In ASCG, I encouraged students to attend the sneak previews and complete the work early. I would often give the students an opportunity for earning A for work in an assignment set that would be a C if completed at the normal time. Thus, students would work extra hard during the first eight weeks so as to complete the entire semesters work early. Many college courses ramp up in the second half of the course as final projects are assigned. This balances the students workload over the semester.

Observe also, in the Contract Grading policies, that students handing in assignments on time do less total problems than those handing in assignments late. This increase in the total number of assignments required discourages lateness and procrastination. It serves the same purpose as a lateness penalty would in a conventional grading system.

Self help books on the subject of procrastination [Burke, 83] often advocate rewards as each phase of a project is completed. I design such rewards, encouraging positive reinforcement (bonuses) into the grading structures. The students are encouraged at receiving them, in contrast to the penalties often given out for lateness under conventional grading policies. (Students are rewarded for earliness; in contrast, the usual procedure in a conventional grading system simply penalizes for lateness.)

I advocate the use of pass-fail exams; these can also be repeated by the student if needed. These serve as a deterrent to cheating on the homework assignments. This feature of the contract grading mechanism is often a major factor in a student choosing this method of grading. Students prefer to have the additional control over the grade afforded by a contract grading system.

The interested reader may be concerned as to the administrative workload. It does take some more time for the professors and students to keep track of the multiple choices and options in the ASCG and GPBCG systems. However this is offset by the fact that all grading is Boolean--I simply have to determine whether a given assignment meets specs or not. We avoid the time that would be spent in a conventional grading system to determine how much partial credit to give for an incomplete or partially incorrect solution. Also, determining the final letter grade at the end of the semester is far easier under these approaches. The requirements for a B are so different form a C or an A that I can quickly determine which grade a student is trying for and deserves by inspection. No averaging, calculators, or spreadsheets are required.

Conclusions

Students vote for contract grading with their feet. In the last three years’ classes, I have given the students a choice between contract grading system as described above OR a conventional grading system. In each of these classes, at least 90% of the students opted for the contract grading system; even though they are told it will be more work.

I used to require that all students accept being graded under a contract system. One semester, I used a survey after such a class that inquired whether they preferred the contract system to a conventional grading system, or vice versa. (Interestingly, one third preferred contract grading; one third of the class stated they would have preferred a conventional grading style; and the remaining third stated they had no preference.)

On average, students complete three times more assignments than would be assigned by other faculty in the department using a conventional grading system. The students complete assignments that are harder than a professor would give to the entire class under a conventional grading system. A third of the students in my CS3 (Assembler) completed A assignments. These involved difficult activities such as three-dimensional arrays, writing floating point emulators and advanced input/output routines. They would be problems that couldn’t be assigned to the whole class without intimidating a substantial number of the students. Also, one half of the class completed assignments that required the reading of a computer manual and learning some aspect of the IBM assembler language without lecture instruction.

Using contract grading properly, a professor can motivate their students to do more work than would otherwise be possible. By allowing students to select problems, one can have all students challenged to their abilities. I have no reason to believe that the contract grading provided a grade more meaningful to the employers of our graduates. However, when one of my A students approaches me for a recommendation, the special assignments that they did for this grade give me plenty of ammunition by which to distinguish them.

I compared the number of assignments completed in my class with a companion class. In my class, we used ASCG based contract grading--students were given the option to choose a conventional grading system. 15% of the students chose the conventional grading system. The other class used a conventional grading scale and ten assignments. Students had to submit all ten assignments working in order to pass the companion class.

That class was taught by a professor who had won two teaching awards. Thus, any favorable difference in my class has to be assumed to be due to the grading system rather than the quality of teaching in my class.

The class using contract grading had an average of 14.33 assignments completed per student (s.d.=10.1). The class using conventional grading had a mean of 9.375 (s.d.=2.459).

This showed that contract grading achieves a greater number of assignments per student. (p < 0.05).

The bottom line is that part of a professor’s job is to motivate our students to do their best possible work. A grading policy is one of the tools that we use to achieve that objective. Properly administered, contract grading systems exceed the motivational power of any conventional grading system.

Fletcher, J. K. (1988). The Correlation of Grade Point Average to Coop Work Performance of Business Undergraduates, Journal of Cooperative Education. 25(1). 44-52.

Fraser, J. A. H. (1990). Student Reaction to Contract Grading. Conference of Atlantic Educators, Halifax, Nova Scotia, Canada, ED328196.

(self-citation) Landamatics in Teaching Computer Programming. Submitted to Journal of Computer Science Education.

Samson, G. E., Graue, M. E., Weinstein, T. and Walberg, H. (1984, Summer) Academic and Occupational Performance: A Quantitative Synthesis: American Educational Research Journal, 21(2). 311-321.