Part I of the Writing Program Assessment

Writing in World Views

Introduction

Context

The Writing Program at Willamette University poses interesting and significant challenges for assessment. While students all take a writing-centered First-Year Seminar, World Views, and all face three paper assignments with identical due dates, they respond to very different assignments, reflecting different classroom emphases in discussion and direction, as well as the disciplinary dispositions of their instructors. Moreover, the assignments they complete range widely in length (anywhere from three to ten or more pages) and in genre (primarily thesis-driven essays, but also frequently with options to write dialogues, scenes, speeches, letters, even poetry). Direct assessment of writing—that is, assessment based on writing rather than on multiple choice exams and the like—and assessment of writing actually prepared for class—rather than for an impromptu timed writing exercise on a uniform topic—mean that evaluators are called on to evaluate each piece of writing and each portfolio on its own merits as the work of a first-year student at Willamette University rather than as an approximation of some exemplary response to a uniform prompt.

The faculty who did the evaluations of the assessment reported below represent a broad cross-section of the Willamette University faculty who all teach writing and who regularly read and respond to student writing. Because Willamette does not teach a first-year composition course in the English Department, as most institutions do, they do not have long years of freshman comp experience or training. They come from disciplines across the campus, some with experience teaching a First-Year Seminar, though not necessarily the one students in this assessment took, some with no First-Year Seminar experience at all. But they will all participate in the continuing writing instruction of these students.

The Assessment Plan

The Writing Program Assessment Plan (approved spring 2002) calls for a five-year cycle of review. The “Assessing Writing in World Views” is the first year’s component. According to the plan, it calls for a review of student writing and of the curriculum. Specifically, we seek to assess students’ success in achieving writing program goals:

1. Students understand that diverse purposes call on diverse processes for writing and become flexible in choosing processes appropriate to the purpose. They recognize a variety of purposes for writing.
2. Students recognize the demands of a variety of readers and develop ways to adapt their writing to meet the needs and expectations of diverse readers. They learn to accommodate the needs of readers.
3. Students gain comfort and facility in writing in a variety of forms. They learn that organizational patterns vary with purpose, readers, and materials, that some forms are clear and relatively inflexible while others are quite loose.
4. Students respect their readers’ expectations for evidence, explanation, and argumentation.
5. Students write in fluent, precise, competent, English in their formal writing; when writing in a foreign language class, they learn the conventions and standards for writing in that language.

And we seek to assess the effectiveness of assignments and pedagogical practices in helping students achieve these goals.

Beginning 2002-2003, on a five-year rotation, the Writing Program Advisory Committee (WPAC) will oversee the review of segments of the writing-centered curriculum and student writing.

• Year 1 (2002/3): First-year Seminar,World Views
• Year 2 (2003/4): Senior seminars,senior theses
• Year 3 (2004/5): Other gen ed/pre-major (100 and 200-level) writing-centered courses
• Year 4 (2005/6): Courses at 300 and 400 level (excluding senior seminars and theses)
• Year 5 (2006/7): Individual portfolios of students entering 2001 (see below)
• The Director of the Writing Center will be responsible for organizing a review of the writing samples collected each year:

1. developing a rubric for measuring the samples against the desired outcomes,
2. recruiting a diverse group of faculty members to read and rate the samples,
3. making arrangements for reading the samples (training materials, refreshments, stipends, etc.),
4. collating, analyzing, and reporting to the faculty on the results.

The Director of the Writing Center will also report on the match between program goals and curricular offerings in each of the course categories, Year 1-4 in the review rotation. This will NOT be a review of each individual course. These results will also be communicated to the faculty.

In Fall semester 2002 (and again in Fall 2007, etc.), the Director of the Writing Center will identify a random sampling of 50 entering students to participate in a longitudinal portfolio study of their development as writers. These students will be asked to save copies (electronically) of their writing, identified by course/instructor/date, for each writing-centered course they take. They will be invited to submit to their portfolio papers from other courses as well. The Writing Center director will monitor their enrollment in writing-centered courses and remind them to submit copies.

Methods

During the Fall semester 2002, the Director of the Writing Center identified the “50 Willamette Writers of ‘06” and collected electronic copies of the papers they wrote for World Views and other courses. 42 of the 50 submitted all of their World Views papers (in one student’s section, only two papers were assigned). She also collected copies of individual World Views instructors’ three formal writing assignments: 12 faculty submitted all three assignments, 2 submitted two of the assignments, 1 submitted 1 assignment; 8 submitted no assignments. In December, 14 (of 23) World Views faculty completed an online survey of their teaching practices in the World Views course.

In January 2003, a group of ten faculty read 36 portfolios and rated them according to a rubric (Att. 1). The group spent about two hours discussing the rubric and reading four papers together, one each from four different portfolios, in order to establish norms. Participating faculty members then read seven or eight students’ portfolios of three World Views papers. They rated each of the papers in the portfolio separately (Att. 2), and then assigned a rating to the portfolio as a whole (Att.3). Each portfolio was read by two different faculty members and the portfolios were distributed so that the same two faculty did not read in pairs across more than two portfolios.

In Summer 2003, a group of 7 faculty members (one of whom participated in January), including the Director of the Writing Center, who teach in the World Views program formed a Working Group on Writing in World Views. The first of their projects was to study the data collected in the assessment project described above. After a norming session, they read the six portfolios completed since the January reading. (Some students did not forward all of their electronic files until they returned for spring semester.) They also did third readings of the portfolios that had received ratings more than one level apart (portfolios that received a 3 from one reader and a 4 from the others were not reread; portfolios that received a 3 from one reader and a 5 from the other were reread.) They read all the assessment materials—the rating sheets prepared by the January readers as well as the faculty surveys and assignments—and considered responses to inform this report which the Director drafted.

Findings

Papers

125 papers were read in all (one portfolio contained only two papers). Tabulating the ratings by both readers—or, in the case of portfolios read three times, by the two agreeing readers—for each of the six criteria, yielded the results summarized in Table 1:

Portfolios

Between the January and June readings, 16 faculty members participated as readers of 42 portfolios, 11 of which required third readings (Att. 4). Their rating of portfolios is summarized in Table 2:

• 20 portfolios rated 4 (Adequate/Acceptable) and above (5, Very Good; 6, Outstanding)
• 13 portfolios rated 3 (Borderline/Fair) and below (2, Weak; 1, Poor)
• 9 portfolios rated 3/4

Writing Assignments

As one would expect, faculty member’s writing assignments varied widely in several ways:

• Nature of the problem writers were asked to address
• Elaboration of the topic with sub-questions and the like
• Defense/explanation of topic, sometimes expressed as “goals for this assignment”
• Inclusion of alternate topics
• Reliance on materials in addition to the World Views assigned texts
• Length of papers requested (Att. 5) * Genre of papers requested (Att. 5)
• Documentation style requested
• Specifications for preparation of papers (some faculty refer to information in their syllabus)
• Calendar for completion of the paper (including first drafts, workshops, conferences, etc.)
• Articulation of criteria for evaluation Nevertheless, the Working Group noted some significant commonalities:
• Dominance of the thesis-driven essay
• Frequency of alternate assignments that include genres modeled in the course readings
• Request for close, critical reading of primary texts
• Relationship between topics and notions of “world view” (implicit or explicit)

Faculty questionnaire

The responses to the faculty questionnaire, including all comments, are tabulated in Att. 6. Surveys were numbered and written comments are identified by the number of the survey. 17 of 23 World Views faculty responded to the survey. Not all faculty members responded to every question, and several questions invited more than a single response.

The responses, of course, vary widely; however, some frequent responses may be noted:

• faculty incorporate and value highly frequent, varied uses of informal writing (writing for learning)
• only about half of the faculty (11 respondents) require peer response to first drafts
• only 7 faculty provide feedback on first drafts to all of their students; others respond only when asked
• only 4 faculty encourage revisions after the graded paper; 3 others allow one revision, 2 allow students who receive poor grades to revise (“especially poorly” or “below B-“)
• faculty recognize thesis, structure, academic conventions, writing as process as significant goals for instruction in World Views

Other relevant data

The Writing Center recorded over 200 World Views consultations. No assessment data specifically addressed the role of the Writing Center in World Views. Students who came to the Writing Center for one consultation almost always came back for another appointment, indicating that they found the Writing Center helpful. Many writing assignments include encouragement to students to use the services of the Writing Center, and in the four or five days leading up to one of the uniform paper due dates, the Writing Center is booked to capacity with consultants working several additional hours to increase capacity.

Discussion

Papers

The tabulations of individual ratings can not be read very exactly. Raters clearly attempted to indicate ratings within each column by placing their checkmarks further to the right or left within the column or even on the line between columns. The tabulations reflect the Director’s best attempt to judge from written comments where the checkmark should be assigned and, when that failed, simply to assign half of the borderline checkmarks to each of the columns straddled.

About half of the ratings for each criterion fell into the “adequate” column, with “strong” and “weak” ratings arranged quite evenly for the criteria “thesis,” “support,” and “error.” There were several more “strong” ratings than “weak” for the criteria “organization” and “style” and several more “weak” ratings than “strong” for the criterion “argumentation.”

21 readings received straight “strongs”; 16 received straight “adequates”; and 8 received straight “weaks.” In most cases, both readers did not give straight marks to the same papers.

Portfolios

Faculty were in general agreement about the rating of portfolios. A total of 16 different faculty members participated in the portfolio reading, all but one of whom had recent (within 4 years) World Views teaching experience. The one who had not taught World Views has taught several writing-centered courses at the first-year and sophomore levels. There were only 10 cases of discrepant readings (i.e., portfolios receiving ratings more than one level apart) and no readers proved themselves consistent outliers.

Ratings also, however, clumped very thick in the two middle levels “4 adequate/acceptable” and “3 borderline/fair” and the split between them “3/4”(n=27 of 42); in fact, while there was only one “2 weak/3” split, there were six “4/5 very good” splits, bringing the total to mid-range portfolios to 34 of the 42 total. This is not an unusual rating behavior: raters tend to see a large middle range. Only one portfolio was rated “weak” by both readers and only one “poor” by both raters. Four portfolios were rated as “very good” and two as “very good/excellent.”

The rating “3 borderline/fair” indicates explicitly that the readers expect the student writer to succeed, with work and commitment, as a writer at Willamette University. In this light, then, the readings are very positive about student writing at the end of World Views. Moreover, in a discussion of the reading and rating process afterwards, the faculty participants noted that it had been difficult for them to distinguish between expectations they hold for first year students and a sort of general expectation for “good college writing”—or even “ideal”—they would hold of any level students.

Finally, a comparison of these 42 portfolio ratings with the Registrar’s record of the 347 final grades for World Views 2002 suggests that the assessment ratings are conservative. Faculty policy requires that World Views grades be calculated at no less than 40% and no more than 60% for writing, with the remaining percentage based on participation. The grade distribution for World Views in 2002 is displayed in Table 3:

Assigned grades, in other words, suggest that World Views faculty assigned significantly more “very good” and “excellent” ratings than the faculty raters did. It would be hard to imagine any grading scale that wouldn’t label A’s excellent (45.5%) and B+’s very good (22.5).

Assignments

World Views faculty annually affirm the importance of each participating faculty member designing assignments that reflect their section’s discussions and interests. But faculty also try to advance similar goals for writing and generally to minimize apparent disparities in difficulty between sections. They do so by circulating assignments—both in draft stage and as distributed—to other faculty on the email list and by discussing assignments in the regular Friday faculty workshops and at special lunch meetings arranged by the Director of the Writing Center.

Based on their reading of the assignments, the Working Group on Writing in World Views prepared a statement of Goals for Writing in World Views to modify the overall Writing Program Goals statement and presented it to the World Views faculty at its penultimate pre-semester workshop, 22 August 2003 (att. 7). They also prepared their own first writing assignments as examples of how they would craft assignments to approximate these goals as well as to demonstrate the result of other work they did on designing and presenting writing assignments.

Faculty questionnaire

The World Views faculty documented their thoughtfulness and deep experience in teaching writing in the comments that elaborated their questionnaire responses (Att. 6). They use writing for learning regularly and imaginatively in ways that align with best practices (Q.2). Their statements of values (Q9) align with the outcomes statement for the Writing Program. Although only one faculty member indicated preference for a standard first-year composition course, a few others expressed uncertainty about their effectiveness as writing teachers. More indicated that they teach writing in all their courses.

Two issues demand attention, according to the survey results. First, providing regular occasions for peer response to writing—explicitly to drafts in process—is required by Writing Program guidelines for all writing-centered courses. Yet 3 of 14 respondents did not require peer review of drafts, in or out of class. World Views course coordinators and the Director of the Writing Center need to communicate expectations clearly, particularly to faculty new to World Views.

Second, responses to Question 6 (“How do you provide feedback to World Views students on first drafts?”) indicate that many World Views faculty are responding only to final drafts, even though Writing Program guidelines call for faculty intervention between first and final drafts. That is, only 7 of 14 respondents reported that they provided written comments on all first drafts; another 4 said that they did “sometimes.” (Both the wording in the question and the array of available responses are problematic in Q6: “all first drafts” was meant to refer to every student’s draft; but two of those who said yes to “all” also answered that they provided “written comments at student’s request,” indicating that only students who asked for written comments got them. The choices for response were “often,” “sometimes,” “seldom,” “never” and should have been “3 papers,” “2 papers,” “1 paper,” “never.”) 4 respondents reported that they held conferences with all students to discuss first drafts, but 3 of these were also among the 7 who provided written comments. The Writing Program guidelines are based on a substantial body of research that demonstrates the importance of feedback during drafts, when the student can actually work with it and test it, rather than post mortem when it is most frequently seen as justification for the grade. Furthermore, responses to Question 7 show that 5 of 14 faculty do not allow further revisions (3 of these did provide comments after the first draft), that 3 allow only one paper to be revised a second time, and that 2 allow only those who did “especially poorly.” Thus, students most often apparently do not have an opportunity to work with the feedback they receive on final drafts. As a practical matter, only the first two papers could be revised beyond the “final draft,” since the third paper is due on the last day of class. The course coordinators and the Director of the Writing Center must see that this issue is addressed in a staff meeting before the next World Views seminar begins.

Actions taken in 2003

As a result of their work on the assessment of writing in World Views, the Working Group led a half-day workshop in August 2003 for World Views faculty (Att. 8). They discussed what they had seen in the 42 World Views portfolios and discussed several matters:

• Defining a set of goals for writing in World Views as a subset of the Writing Program outcomes
• Length and genre of papers assigned in World Views
• Clear attention to discussion of the thesis (discovering, developing, complicating)
• Ways to use the required handbook
• Responding to student writing (briefly)
• Recognizing and responding to plagiarism

As a result, the World Views assignments collected by the Writing Center demonstrate some changes in pedagogy:

• Clearer definition of goals to be accomplished by an assignment
• Clearer articulation of evaluation criteria
• Increased attention to the thesis

Writing Center consultants also reported that WV students who visited the Writing Center were very concerned about thesis.

Conclusions

The First-Year Seminar, World Views, is serving students well as a first college writing experience. Students are, on the whole, writing to faculty expectations in World Views, and those expectations are high. If their writing-centered courses continue to be as challenging as World Views, we will expect to see continued growth and maturation in writing over their four years at Willamette.

Faculty are, on the whole, teaching writing creatively, conscientiously, and effectively in World Views, employing a range of pedagogies. They are providing students with a rich writing environment, including many and varied uses of informal writing for learning; assignments which face them with challenging academic assignments that vary, occasionally in genre and imagined audience; practice in developing arguments from sources, through drafts, responding to reader feedback; and instruction in the conventions of academic discourse, including citation and documentation. Friday meetings and occasional brown-bag lunches offer faculty occasions for excellent sharing of resources, ideas, insights, and materials in the teaching of writing, and faculty clearly benefit from the exchanges.

This assessment also shows that there are still conversations and workshops to be had about the uses of peer response in writing instruction and about effective and efficient faculty response to drafts. Because the First-Year seminar, World Views, is such an important component of Willamette’s writing program and because new faculty join the seminar, it will always be worthwhile both to recall and reinforce the basics of curriculum and pedagogy in writing.

Finally, the assessment design seems to be sound. That is, given the “real world” character of the student writing in the First-Year Seminar—assignments arising in the context of a course and different from one section of the course to another, papers read as pieces of writing rather than as responses to a particular assignment—raters were able to use the rubrics developed and to make judgments, if fairly conservative ones. In another round, we might wish to try linking papers to the assignments that inspired them, a practice not engaged this round because of the emphasis on program assessment rather than on individual faculty successes. We might also wish to develop a questionnaire for student assessment of their learning, providing richer description than that provided by the Student Evaluation of Teaching Effectiveness (SETE). And we might include direct questions both to students and faculty about the role of the Writing Center.

The assessment process has already been useful to the faculty who participated as raters, providing an occasion for extended reflection upon student writing. With the publication of this report, that reflection should engage larger portions of the CAS faculty and academic administration.