This page is part of the 2016 Lane website archive, and is presented for historical reference only.

CLO Synthesis Report, 2013-2014

CLO Synthesis Report, 2013-2014

Background for the A-Team Synthesis Report, 2013-2014

Overview of CLO work, 2013-2014

Prepared by Kate Sullivan, CLO Coordinator, with assistance from Christina Howard, A-Team Chair

The A-Team’s work for 2013-2014 was established through work and conversations at the June 2013 AAC&U Institution on General Education Assessment in Burlington, VT.

Participation at the AAC&U conference led team members Sarah Ulerick, Christina Howard, Christine Andrews, Molloy Wilson, and Kate Sullivan to conclude that our strategy for assessment must shift from a focus on individual courses and individual or dyads of instructors crafting small projects for their own use, to a greater concentration on assessment projects tied to program review and/or development of rubrics with disciplinary specificity, connected to signature assignments.[1] Such a focus necessarily involves an alteration in consciousness for the college and for staff, administration, and faculty working within a departmental context.

Assessment Plan for 2013-2014

Before leaving the AAC&U Institution, the assessment team assembled our three-year assessment plan; goals for the first year are to increase visibility and understanding of the strategic direction, “a liberal education approach to student learning,” and increase visibility, understanding, and ownership of our Core Learning Outcomes (CLOs): Think, Engage Communicate, Create, and Apply.[2] 

Efforts to increase visibility have garnered success with various groups across campus, including: our dual credit program coordinators; the head of publicity, Tracy Sims (who has supplied the A-Team with a variety of promotional materials and who has been involved in the creation of CLO banners that will be visible across campus, beginning Winter 2015); and counseling and advising faculty and staff; as well as faculty groups in various disciplines (Art, Speech Communications, Spanish, French, Writing, Biology, Physical Therapist Assistant, and Dental Hygiene).  Although our CLOs were created by a subcommittee of faculty[3] and endorsed by a significant portion of our faculty, we have yet to reach a tipping point on campus where the majority of faculty and staff have sufficient familiarity with our CLOs so as to incorporate them clearly and easily within courses and programs.  It is doubtful that we will reach such a point without a more sustained and intentional commitment from the highest level of administration on campus.[4] 

The RFP: Background and Submission Teams

Faculty Teams:

  • Art: JS Bird (lead), Satoko Motouhi, E. Beyer, G. Soracei, C. Harger
  • Biology: Lisa Turnbull (lead), Christine Andrews, Stacy Kiser
  • Communication: Karen Krumrey (lead), Jay Frasier, Hyla Rosenberg, Barbara Breaden
  • Dental Hygiene: Sharon Hagen (lead), Michelle Cummings, Leslie Clark, Rita Kavanaugh, Jill Jones, Tammy Maahs, Cris Houser, Imy Cully, Tammy Sutton, Vicki Dodge
  • French: Karen Almquist (lead), Valerie Metcalfe
  • Physical Therapist Assistant Program: Christina Howard (lead), Gary Ahearn, Mark Duyck, Brian Wilkinson
  • Spanish: Matt Luke (lead), Bojana Stefanovska, Sylvie Florendo, Roma Cusimano
  • Writing (English): Heather Ryan (lead), Kate Sullivan, Siskanna Naynaha, Aryn Bartley, and Sarah Lushia

In order to prompt a more programmatic approach to assessment, we crafted our RFP to emphasize the communal nature of the projects we were soliciting. We designed the RFP to reflect the varied levels of preparation and readiness that departments and programs had with assessment work and earmarked three levels of funding: A) 22 hours for programs, departments, or division teams to hold conversations about how the CLOs intersect with program and/or course outcomes, work that may also involve development of a discipline-specific rubric; B) 18 hours for mapping CLOs to course outcomes; C) 75 hours for programs or departments that have already developed a rubric and are able to score artifacts using this rubric.

The majority of teams—Spanish, French, Speech & Communication, Writing, Biology, PTA, and Dental Hygiene— applied for funding levels A or B; one group, Art, applied for funding level C, artifact scoring.

We approved all groups that submitted a request for funding, although Christina Howard did work with Dental Hygiene to revise their request to better fit the parameters of the RFP.

Process of Working with Department/Program Teams—the Role of CLO Coordinator

Kate Sullivan, CLO Coordinator, met with faculty teams to discuss their RFPs and to plan their assessment actions; these meetings took place over winter and spring term and extended into summer 2014; additionally, she met with the social science division several times to discuss the role of CLO assessment within the context of various programs. Several key themes emerged from these conversations:

  1. Faculty do not always have a clear idea of how CLOs are manifest within their course and program outcomes, and sustained and engaged discussion is a productive method for helping individuals and groups identify the intersection of course- and program-level outcomes with our general education, outcomes/CLOs and translate the CLOs into appropriate, discipline-specific language and vice-versa (discipline outcomes into CLOs);
  2. Departments are frequently unready to engage in assessment work until they have worked on program review[5];
  3. Departments/Programs/Disciplines that have national standards and/or external groups or accrediting agencies that suggest disciplinary goals have an easier time undertaking program review AND CLO mapping;
  4. Departments with a high concentration of PT faculty have a considerable challenge in undertaking assessment work given that program coherency and a common understanding of course outcomes and our CLOs as they relate to these outcomes may be lacking, especially if faculty haven’t had the opportunity to develop facility with assessment concepts, methods, or terminology;
  5. All faculty believe there is a great need, not only for CD money for assessment work, but for actual reassignment time for program leads or coordinators to undertake this work.

The Need for Department Conversations around CLO Translation and Program Review

Faculty do not have a clear idea of how/why the CLOs are manifest within their disciplines. This fact was, again, revealed to us during our presentation and workshop at Spring Conference, 2014, “Engaging with CLOs,” as several faculty in the audience at the presentation/workshop made comments such as, “my course doesn’t focus on critical thinking,” or “my students don’t create anything.” Upon closer analysis, faculty realized that their courses may, indeed, focus on either of these CLOs (or others), but disciplinary language differed from the phrasing in our CLOs.

Thus, much of Sullivan’s work, following the RFP, with faculty involved rich and deep conversations about the intersection of CLOs within a given program’s courses and outcomes. In fact, Speech, Spanish, Biology, and French all used their funding to work on aligning course outcomes/outlines with our CLOs and to create contextualizing and supplemental materials to assist with program coherency and, eventually, to create signature assignments. Specifically, some of the work accomplished marked a shift from the initial plans articulated by faculty groups since conversations with Sullivan revealed that the faculty groups were not yet ready to undertake rubric development. As a case in point, we would like to describe what happened with three programs, Speech & Communications, French, and Spanish

Speech & Communication

Initially, Speech requested 18 hours to map the CLO “communicate effectively” to their class, COMM 100 (formerly SP 100), given that they’d already created a rubric for “communicate” the previous year. During the course of conversations between Karen Krumrey and Sullivan, however, Speech faculty indicated that there are a number of steps that need to be undertaken before the department is ready to map CLOs to their outcomes. First, a number of the outcome statements in the course outline were not actually objectives but rather amorphous descriptions of course content. Additionally, the course outline was rather vague about the assignments/projects that students would complete and by which they’d be assessed.[6] Krumrey indicated that she thought it prudent for Speech and Communication faculty to work on revising their course outlines and setting up parameters for student work before mapping CLOs against said course. Specifically, before Speech/Comm faculty can move forward with assessing student work, they need to clarify their own objectives/course outcomes and work on supporting documents that will articulate course parameters and goals—in other words, they need to undertake program review.[7]

Ultimately, what Speech/Comm faculty accomplished in 2014 was to hold a series of conversations about assessment goals (see this report), the revision of the course outline for COMM 111, and the creation of surveys for faculty and students to be given towards the end of Fall term, 2014, to assess visibility, engagement, and student proficiency with course outcomes (see the section on Spanish for a more thorough explanation of the survey instrument). Through the process of creating these surveys, Speech/Comm faculty were able to revise their course outcomes and better articulate goals for student learning. Beginning fall 2014, faculty will be submitting revised course outlines to the division and on file with the college.

Speech/Comm faculty expressed some frustration with what they’ve perceived to be a lack of coherent direction in campus-wide assessment efforts. Specifically, they identified earlier calls for rubric creation not tied to signature assignments nor connected clearly to course and program goals. Such rubric creation did not, they observed, tie directly into productive program review and/or faculty conversations about effective teaching and student learning. Their observations dovetail with JS Bird’s 2012 report about the difficulties faculty faced in using the college-level rubrics for "Communicate Effectively" to score artifacts. According to Bird, the generic rubric did not necessarily serve faculty needs within a disciplinary context, and scoring by individuals who lack disciplinary knowledge was challenging.

French

The efforts by French faculty were restricted to the involvement of two individuals—Karen Almquist and Valerie Metcalfe—due to enrollment shifts and loss of PT faculty positions Spring 2014; other French faculty members were working elsewhere and unable to commit to the assessment work. This scenario in French was repeated elsewhere on campus and reveals the challenges facing a school with a predominately PT faculty: LCC’s use of peripatetic faculty limits the thoroughness of our assessment efforts and the widespread involvement of all faculty within a given department or program.

Conversations with French faculty revealed some confusion about the role of CLOs and mapping them to course outcomes. Initially, there was some confusion—do all of the CLOs need to turn up in every class? Does every course outcome necessarily need to map seamlessly to a CLO?[8] In both cases, the answer is, “no,” but faculty confusion is instructive given our attempts to raise visibility about and consciousness of, the CLOs. Obviously, we still have additional work to accomplish.

The bulk of French’s work focused on mapping CLOs to the department’s primary documents: the course information sheet for FR 101 and a rubric for use in identifying signature assignments in the first-term class. Additionally, they submitted their five-year assessment plan (that began in 2010 and ends in 2015). This plan is available on the 2013-2014 Reports page of the Assessment website.

Spanish

Spanish faculty met with Sullivan multiple times winter, spring, and early summer of 2014. Like French, Spanish faculty initially had some confusion about the intersection of CLOs with course objectives, and a good portion of the conversations with Sullivan involved translating CLOs into the course outcomes for SPAN 101. In particular, faculty indicated that they see the outcomes—Think, Engage, Communicate, Create, and Apply—as inextricably intertwined within language learning.

Consequently, there was some concern about what they considered the artificial separation of CLOs into discrete rubrics. In other words, the learning objectives for a particular Spanish assignment do not easily separate out “thinking critically” from “applying” methods or skills to a given task. Thus, Spanish created a combined CLO rubric to be used in evaluating one section of a shared exam. This two-part rubric (Rubric part A and Rubric part B) —"Think, Communicate, Apply"—will be shared with faculty and used in the evaluation of the dictation portion of one (or all) of the five shared exams in SPAN 101.

A second major accomplishment involved the creation of faculty and student surveys to be used towards the end of Fall 2014. These surveys will be used as a form of indirect assessment to evaluate perceptions about student learning in SPAN 101. Students will be asked to evaluate two issues as they relate to learning outcomes: to what extent they were given the opportunity to learn a given outcome; to what degree they feel they’ve mastered this outcome. Faculty will be asked to weigh in on three issues: how much they value a given outcome; how well they feel they teach it; how well they feel students gain proficiency with this outcome. Ideally, results from these surveys will allow faculty to determine to what extent student and faculty perceptions match and to adjust teaching and/or curriculum accordingly.

The Role of External Agencies/National Groups in Assessment Efforts

Writing

In the English department, a group of six faculty[9] worked on two main projects: developing supplemental documents to articulate course culture[10] and objectives to students and to faculty and a rubric faculty may use individually to assess end-of-term essays in WR 115 and that also may be used in the end-of-year artifact assessment project undertaken by faculty teams each September.

Like Spanish and Speech/Comm, ENG faculty realize that program review and course consistency are necessary prerequisites for undertaking assessment. The Writing program is also fortunate to have local and national standards to look to for assistance in articulating course and program goals. Indeed, faculty on campus who have been most successful with undertaking assessment work—many of the C/T programs such as PTA and Dental Hygiene—have both external accrediting agencies and national standards to help them devise outcomes and conceptualize program review/course progression. In particular, writing faculty can draw on documents published by the Council of Writing Program Administrators (CWPA) for guidance, as well as documents published by OWEAC, the Oregon Writing and English Advisory Committee, which also draws on the work of the CWPA and the National Council of Teachers of English (NCTE). Key figures in the English department are also active in regional and national conversations about writing assessment, another factor in the department’s relatively advanced status in program assessment.[11]

However, the CWPA is in the midst of updating their recommendations for first-year writing, and ENG has postponed their own revision of outcomes until they can first look at work by the CWPA, which also will affect OWEAC's recommendations for writing courses.

Biology

Faculty in Biology undertook CLO mapping work, focusing on “engage” as it relates to PULSE (Partners for Undergraduate Life Sciences Education) Standards, in the context of BIO 100. Lisa Turnball initiated a crowd-sourced mapping activity in the Science Division Lounge. This low-stakes exercise allowed broad-based involvement, as faculty used sticky notes to indicate what kinds of activities they are using with students to meet course outcomes. This mapping project dovetailed with two key organizations/events: the National Science Foundation’s “Vision and Change” Project; a presentation of LCC Biology faculty’s CLO mapping and program review presented at the Northwest PULSE National Conference for Science Educators in Spring 2014. PULSE and NSF Educators are, like LCC faculty, focused on clearly articulating learning goals for science students and developing clear objectives for science educators. Thus, our local faculty were able to draw on national conversations and receive both native and national feedback (via googledocs) on their mapping activities.

Biology faculty plan on using a survey to determine how faculty can implement the NSF’s Vision and Change goals in the BIO 100 classroom (and beyond); they will also be working on developing a new learning module that will encompass both LCC’s CLOs and the NSFs Vision and Change Science and Society’s competencies for life science students.

PTA

The Physical Therapist Assistant (PTA) Program mapped the CLO “Apply” to first-term course outcomes. The initial goal is for students and faculty to have an increased understanding of how and why CLOs are important and where a particular student is in his/her development of proficiency against the specific CLO. Under the guidance of Howard, PTA faculty discussed (both F2F and digitally) what “apply learning” means within the context of PTA 100, 101, and 101L. Howard supplied a worksheet to help faculty with this conversation, drawing on the “Apply” rubric created by Howard and Brian Wilkinson in 2012-2013, with the goal of examining and evaluating assignments as they relate to the "Apply" outcome. In April 2014, faculty met to discuss their understanding of how "Apply" fits into course outcomes and class assignments. Both PTA 100 & 101L now have clearly articulated linkages between CLOs (“Apply”), course and program outcomes, and assignments, and they’ve increased visibility of this outcome on both syllabi and assignments. Additionally, faculty have increased their understanding of how “Communicate Effectively” is manifest in the curriculum as well.

Faculty used Bloom’s taxonomy to delineate between knowledge-based assignments and assessments and higher-order critical thinking skills (our CLOs), as well as the spider-web mapping tool created in 2012 to help facilitate our understanding of the Degree Qualifications Profile.

A really positive take-away from PTAs assessment project is that they now have a good model and process for future CLO mapping, which will also increase program consistency and coherency. Another key outcome of the project is that faculty who participated in these activities definitely have a clearer understanding of the value and role of CLOs for student learning.

Dental Hygiene

Dental Hygiene’s assessment work has been undertaken in conjunction with program review processes and objectives that are dictated, to a large degree, by the Accreditation Standards for Dental Hygiene Education. During this assessment work, they met four times; first, ten faculty members reviewed the LCC CLOs and examined Oregon Institute of Technology’s program assessment for their Dental Hygiene program, which was used as a model/inspiration for LCC conversations; next they developed a student self-assessment rubric of CLO attainment for DH 221B, which was used in Winter 2014; these self-assessments will be used in both DH 221B and DH 222B.

At the next meeting, faculty discussed and mapped the “Communicate” CLO across courses. The third meeting focused on end-of-program assessment measures and accreditation standards and the documents that need to be produced for accreditation, which are based around the Commission of Dental Accreditation (CODA) standards.

The last meeting focused on documents that will be used for formal accreditation—faculty assessments, patient satisfaction surveys, chart audits, and employer surveys of DH graduates of LCC within the past 7 years. Additionally, faculty developed a plan to integrate CLO assessment work into the formal accreditation process by including the following beginning the Fall of 2014: an exit interview, a graduate survey, personal competency assessment, all indirect assessment measures that will all be based around a developmental rubric for the CLOs. This rubric will instruct students to evaluate themselves as accomplished, proficient, developing, or beginning against each CLO, a process that will also raise visibility and understanding of the CLOs.

Artifact Evaluation and the Challenges of Contingent and PT Faculty

Art

Under the leadership of JS Bird, Art faculty endeavored to undertake artifact scoring, using the “Create” rubric that they crafted in 2012-2013. In fall of 2013, faculty who teach a variety of Art classes began collecting artifacts for future scoring and met to plan assessment work. Unfortunately, LCC’s budgetary challenges resulted in a radical reduction of art courses spring term and a concomitant loss of PT faculty members to participate in artifact scoring, so the direct assessment project they had envisioned had to be postponed. To complicate matters further, Bird, the faculty lead for the project, is on sabbatical for fall term 2014, which may mean that the project cannot move forward until Fall 2015, unless another faculty member is able to step forward and helm the work.

Conclusion

2013-2014 marked a period of faculty interest and enthusiasm for assessment projects, a phenomenon that we attribute to all of the groundwork laid by the work of the A-Team over the past seven years. We are excited about faculty interest and commitment and believe that all programs and departments that were funded for assessment work this academic year are poised to continue their projects and further our campus work. This is a tremendous achievement and marks the development of a paradigm shift in faculty attitudes and thinking about the role of assessment on campus.

However, our optimism is cautious, marked by concerns about institutional support and the structural conditions that are necessary to facilitate on-going assessment work. What follows are our concerns and observations.

Faculty have some confusion about the reach and authority of A-Team, which is fundamentally a mentoring group, not a group with the power to shape policy or compel faculty to produce work. Our lack of power, though, is symptomatic of the institutional gap in our assessment work. To date, we have had a series of faculty coordinators who are, in effect, cheerleaders for assessment work across campus. Their enthusiasm and efforts have been invaluable in raising faculty awareness about the value of a culture of assessment—indeed, Fall 2013 marks the first time that there was little, save in pockets on campus, resistance to the value and work of assessment—however, without more institutional support, including administrative authority, we will be unable to establish and ensure institutional and systematic assessment across campus that must be, necessarily, threaded throughout programs—faculty who are already interested in assessment are willing to engage with RFPs and projects, and incrementally, new faculty are expressing interest in taking on this important work. Nonetheless, a significant portion of faculty on campus remain ignorant (or suspicious) of both our CLOs and what meaningful assessment looks like.

The experiences of French, Spanish, and Speech/Communications point to the need for systematic program review that involves a significant number of faculty teaching in any given program or department. Absent such program review, we will be unable to develop thorough assessment projects (move on to actual direct assessment of artifacts).

In several departments/programs, the ratio of PT to FT faculty is a real impediment to assessment work, as is lack of reassignment time for program leads. The assessment reports of French, Spanish, Speech & Communications, Biology, Dental Hygiene, and English all indicate that lack of reassignment time and/or resources impede persistence toward meaningful work.[12]

Finally, the recent increase in course caps has left a number of faculty now unable to participate in assessment work.[13] Given the tenor of much of the conversation with faculty—how overtaxed and stretched beyond their limits they already were and are—we have concern about the level of engagement of faculty in departments and programs targeted for increased course enrollments (Writing, Biology, Communications).

As a final note, we want to clearly state that we have moved from the time when faculty attitudes and lack of understanding functioned as the primary impediment to the college’s assessment work to today, when our situation is that material conditions—lack of reassignment time, an absence of infrastructure or support staff, lack of communication and support from administration—are the obstacles to systematic and effective assessment.

[1] For more explanation of this shift, please refer to the A-Team’s Six-Month Progress Report.

[2] For an explanation of the process by which we arrived at these Core Learning Outcomes, please consult Christina Howard’s 2013 Assessment Projects Summary.

[3] The project of creating new CLOs to replace our earlier “Core Abilities” was helmed by Barbara Breaden in 2011-2012.

[4] One repeated message that the A-Team group heard at the AAC&U Institution was that for assessment to become truly integrated into a given college’s culture, the mandate and efforts must stem from campus leadership. Indeed, many of the schools we encountered at the Institution indicated that they have an office of Assessment that is dedicated to FPD and support of assessment efforts across campus. Such a department necessarily supplies practical and pedagogical resources for faculty, departments, and programs undertaking assessment efforts. In LCC’s case, given the exigencies and organization of community college departments into disciplinary groups that often lack chairs or faculty leads, program-level assessment is particularly challenging.

[5] This phenomenon has already been noted by the A-Team in the 2006-2007 Assessment Report, in which we explicitly called for program review.

[6] For instance, in COMM 111 (Introduction to Public Speaking), heretofore, there have been no requirements that students deliver an actual speech, and in some classes, students will deliver no speeches orally, while in other sections, students will deliver ten speeches. Thus, part of Krumrey and Sullivan’s conversations involved how to set up reasonable parameters for student work and how to develop signature assignments.

[7] As an initial step, Krumrey worked on the course outline for COMM 111, and with input from Sullivan, revised outcomes for this class, to be presented to students and faculty in a Fall 2014 survey about course goals.

[8] PTA faculty engaged in a series of similar conversations and expressed initial confusion about the relationship of CLOs to course outcomes, and part of their assessment work involved reaching the following conclusions through an organic process: not all CLOs will show up in every class; not all course outcomes will map to CLOs, and this is okay; some course outcomes involve multiple CLOs and any rubric designed to assess said outcome must also include multiple CLOs.

[9] Initially, the members of this group included Sarah Lushia, Aryn Bartley, Siskanna Naynaha, Heather Ryan, Will Fleming, and Kate Sullivan. Fleming, a PT instructor, had to step off the project in the spring, given other employment responsibilities outside of LCC. Ryan, another PT instructor, was able to continue her work since she was also employed by the college as one of the primary technical writers on our 2014 Accreditation Report and had consistent employment at LCC. ENG has a significant portion of those working on this project have FT status. Other programs, such as Art, have struggled with the ratio of FT to PT faculty and had their assessment projects stall because of this fact, losing momentum and continuity when key PT players no longer held positions on campus during a given quarter.

[10] These contextualizing documents include descriptions of the revision process, critical reading for college, attendance expectations & policies, and contextualizing information about peer workshops. The primary audience for these documents is students, but a secondary audience is new faculty, with the goal of familiarizing themselves with the culture of the department and the composition classroom. These documents will eventually be housed on the department webpage, available to the general public.

[11] For instance, Lushia, Naynaha, and Sullivan presented on LCC’s assessment efforts at the 2013 TYCA (The Two-Year College Association, a subsidiary organization to NCTE) Conference in Seattle. They focused, respectively, on course-level, program-level, and campus-level assessment of CLOs at LCC.

[12] English is an interesting case study. Unlike many of the other programs, Composition does have a program lead, Dr. Naynaha, who relied on a LETS worker to collect and organize the database of writing artifacts for use in English’s annual writing assessment project. In 2013-2014, the program lost their LETS worker, and the collection process has stalled as a result. This situation underscores the degree to which assessment work is facilitated not only by faculty interest, knowledge and engagement, but also by actual resources such as digital storage space and a student worker who managed the database and undertook the de-coupling of artifacts from identifying marks, ensuring a degree of anonymity that facilitated participation from reluctant English faculty.

[13] In what is perhaps a most tragic irony, course caps have been increased in three key departments—Biology, Communication, and English—poised to undertake more systematic assessment; faculty from these areas have expressed frustration that their increased workload will interfere with their ability to continue the assessment work they have recently undertaken.