Manifesto: History

History:
How Portfolio

Updated: 05/31/07

A Brief History of First-Year Composition Student Competency Evaluations
Compiled and Written by Marlene Chinello

Over the past twenty-four years, the English Department has striven to find a more accurate, more consistent, more useful, less time-consuming method of evaluating the success or failure of first-year composition students. This has taken us through approximately seven major changes, usually occurring a year or so after the appointment of a new Director of Composition or new State guidelines.

Note from the author: Since this brief report is a recollection based on memory, not records, the dates might not be entirely accurate. Also, the interpretations are based on our experience. If anyone who was present at the time has a different interpretation of this, he/she is welcomed to add corrections, clarifications, and further commentary.

Stage One: 1977 through 1986
Students were given a pre- and post-test, similar to the GWPE, which was read blink, in that the name of the student and whether the sample was the pre- or-post test was kept secret. The students wrote two essays, one analytical and one personal, within a 45 minute time limit, based on two questions or prompts. The advantage to this was consistency in that all students responded to the same sets of questions and could be evaluated comparatively quickly and accurately. The disadvantage was that the test itself was not a reflection of the process approach, and thus it was not an effective measurement of the goals we set for the course. If students were encouraged all semester to make multiple drafts and work in stages, then it seemed unfair to give them a timed, one-shot, pass-or-fail test. Consequently, the evaluation method was changed to the Writer’s Portfolio evaluation.

General Requirements and Process for Writer's Portfolio Evaluation:
Students are required to submit three three-to-five page papers at the end of the semester to a “blind” reading where the students’ identities are kept secret by use of a code number. Two faculty readers score each portfolio, and two of the three papers need to pass. If there is a significant discrepancy in the scores, a third, highly experienced reader arbitrates. If the student passes, he/she has satisfied the First-Year Composition (FYC) requirement; if the student fails, he/she needs to take another writing class, English 200, and resubmit a passing portfolio. Except for Stage Five, the second reader has no knowledge of the first reader’s score.

Stage Two: 1986 through 1989ish
The first Writer’s Portfolio included one narrative and two expository papers, including any thesis-driven papers, personal essays, and mode-based papers such as illustration, compare/contrast, problem/solution, persuasion, literature analysis, etc. Groups of three or four instructors would evaluate class sets of papers on a four-point scale, with the Director of Composition being the final arbitrator for scoring discrepancies. The advantage to this was that it allowed for a variety of assignments and reflected the students’ ability to use multiple drafts to achieve a polished product. The disadvantage was that sometimes the combination of passing papers did not prove analytical ability or show enough evidence of critical thinking, since the passing papers might be a narrative and an illustration, for example.

Stage Three: 1989ish through 1993/1994
This stage was similar to the one above except that the content of the portfolio changed: one narrative, one analytical paper, and one free-choice paper, with the requirement that the analytical paper be one of the passing papers. The free-choice papers could be any class writing assignment that was polished, including all the papers described in stage two, creative pieces (including long poems, extended journal entries, short stories, and literature assignments), and pure description (also called “saturation papers”). The advantages to this were that the department insured that analytical ability and critical thinking were in evidence, the choices reflected more “real world” writing, and the large variety of options allowed us to draw on a particular student’s strengths. The disadvantages were that Stage Three portfolios still didn’t show sufficient evidence of the student’s critical thinking, and the large variety of assignments proved challenging to evaluate. It was hard to agree upon common criteria and interpretation standards for evaluating all the different types of papers. The Director had to settle some discrepancies, for the small groups might contain people whose values and standards were very different.

Stage Four: 1994 through 1995
This stage was similar to Stage Three except that the reading groups expanded to teams of eight, with a Team Leader who was to arbitrate. Instead of the four-point scale, we moved to a six-point scale to make the assessment rubric one that could offer a greater differentiation between papers. Also, instead of an evaluator reading a whole class set of student papers, the portfolios were randomized. The portfolio consisted of two analytical papers and a narrative, and one of the analytical papers must pass. There were fewer re-reads. This procedure was working well, but in the interest of doing an even better job, we decided to move into stage five because some of the new instructors had a hard time explaining to their students why a particular paper didn’t pass.

Stage Five: One Semester Following Stage Four in 1996
The teams and portfolio requirements remained the same, but a new rubric was used. It consisted of a full page check-sheet, separated into six main areas--focus, organization, sense of audience, editing, etc.--with four or five sub-topics that could be checked. These were graded Satisfactory, Unsatisfactory, and Weak. Because the new rubric was so long and complicated, columns were provided for a reader to make annotations because it would be hard to collate all the data on three separate sheets. Two Us (Unsatisfactory) in any main category would fail a paper, but Ws just noted that the area needed to be improved upon. There was also an area for comments. The advantage to this was that an instructor had confirmation that a particular area of a paper needed work, and a student had a better understanding of what exactly he/she might need to do. The disadvantages were that using this rubric took two to three times longer than a holistic reading, and too many students still didn’t understand the comments. A reader could also see how the prior reader(s) graded the paper and might be influenced; even though the first reader was supposed to fold the evaluation sheet to obscure the annotations, some forgot, and it was hard to avoid seeing the commentary section, even though we figured out a better way to avoid this during the actual reading. Furthermore, while this was a decent tool for the midsemester reading, it was of questionable value for a final reading, since the students didn’t revise, and there were a lot of inconsistencies in evaluating the criteria. One reader might give a particular area a W, another a U. We decided to refine this and give it one more shot, but just for the early reading.

Stage Six: One or Two Semesters Following Stage Five in 1997
We used the above method for the early reading and gave codes for remedy. Using a slightly refined rubric, we experimented with a couple of codes, one only for the midsemester reading. For instance, a 2A code might mean that all the paper needed was editing; a 2B meant minor revision and editing (for these, if the student solved the problem to his/her instructor’s satisfaction, the paper would go into the final reading as having been passed). The score of 1 meant the paper passed. A 3 meant the paper needed serious, significant revision and needed to be resubmitted to the Director of Composition upon such revision for reassessment prior to the final reading. A 4 meant that the paper topic was inappropriate for an academic audience and should be abandoned; this was very rare. The final reading was on the six-point scale we used in Stage Four. The advantages here were that the students were fairly clear about what they had to do, and a high degree of accountability was afforded to the program, but the Director of Composition was burdened with reading and evaluating a large number of papers. Another disadvantage was that the readings were incredibly laborious and excessively time-consuming, with each portfolio reader taking between 30 minutes to an hour to assess each portfolio, and reader fatigue somewhat negated the accuracy of the results for the papers which were read later in the day. Because of this, we decided to redesign our new six-point rubric to include more student-accessible data, and we increased norming sessions so that we, as instructors, could more clearly explain to our students our evaluation criteria and standards. However, the Director of Composition wanted even more evidence of critical thinking and analytical ability, so we changed the analytical requirement for Stage Seven.

Stage Seven: Spring 1998
The portfolio composition was the same, but both analytical papers were required to pass. The advantage was that analytical ability was assured if a student passed, but the narrative became devalued, since it didn’t matter if it passed or not.

Stage Eight: Fall 1998 through Spring 2001
Because of the new State requirement to eliminate remediation classes, remedial students are required to take an accelerated class which incorporates remediation into the standard Freshman Composition class. We felt that in order for these students to be able to pass the analytical requirement, we would need to focus more on expository and analytical papers, so the 100I course shifted term paper assignments to mostly all analytical writing, though other types of writing are done less formally. The English 100 course probably has a higher degree of flexibility. We score as one large group, with all papers randomized, using “invisible pens” to insure that there is no prior reader’s influence on the second reader, which was only a problem with the larger rubric. In addition, we began scoring the entire portfolio as a whole rather than the sum of its parts, and we acknowledged the importance of the portfolio cover letter as a self-awareness and self-reflective tool. The advantages to this are that standards and evaluations are much clearer, accountability is excellent, and with two experienced arbitrators acting as Table Leaders, the Director of Composition is not burdened. With all the papers evaluated being analytical, students have a better opportunity to pass the analytical requirement. The disadvantage is that there is limited variety which does not reflect “real world” writing challenges, and students who might have good skills in other types of writing only don’t get a chance to excel.
(End of Ms. Chinello's text.)

Continuing Improvements:
Stage Nine: Fall 2001 through Spring 2002
We adjusted our view of the assessment portfolio, treating it as a body of work rather than the sum of its parts. Specifically, we lifted the individual page limit for portfolio submissions: instead of requiring three three-to-five page papers, we now ask that each portfolio contain three pieces of writing and a cover letter and that that those four pieces add up to ten to sixteen full pages. We also reinstituted the free-choice submission. In these ways, we believe we honor and validate student choice, making sure students maintain authority over their learning and benefit from the responsibility associated with that authority.

Stage Ten: Spring 2002 through Fall 2004
We maintain the same six-point scoring guide, but instead of performing two separate readings during the marathon scoring session, each instructor now acts as her/his students' as first reader. In this way, we perform second and (perhaps) third readings--not first, second, and (perhaps) third readings--during the marathon session. Each portfolio still benefits from one blind reading, and if teacher and reader scores don't jive, very experienced readers settle the discrepancies.

Stage Ten A: Spring 2005 through present
We maintain the same six-point scoring guide and scoring process as described in Stage Ten, but now the scoring guide includes a 0 score and no longer contains descriptors ("Strong," "Developing," etc.).

Effective Fall 2005, poetry and fiction are ineligible for portfolio submission.
Effective Spring 2006 (?), the minimum page requirement increases to twelve full pages.
Effective Fall 2007, the minimum page requirement increases to thirteen full pages.