From Chaos to Consistency: A Human-Centered Framework for AI-Supported Assessment at a Mission-Driven Liberal Arts College

Synthesis: Ravesloot, Copeland, and Pryor (2026) report a Spring 2026 pilot at a small Catholic liberal arts college in the Midwest that used a generative-AI-supported Assessment Wizard to help faculty complete program-assessment documentation during a structured Education Day. Grounded in Catholic social teaching and human-centered AI principles, the pilot framed the work around three principles — human judgment as non-negotiable interpretive authority, Ethics of AI-augmented evidence, and formative human-AI partnership. Survey results from the pilot cohort (N = 10, with item-level valid responses varying) showed strong endorsement: every participant selected the highest response option for future use (M = 5.00, SD = 0.00), and the process was rated most helpful for thinking carefully about evidence of student learning (M = 4.67). Thematic analysis of nine participant accounts identified five themes, including efficiency and cognitive Scaffolding, human-in-the-loop validation, platform reliability and trust, data governance, and institutional support infrastructure; a negative case in which one participant spent about six hours on a separate program-review assistant and then finished the task in a personal ChatGPT account in about two minutes exposed governance vulnerabilities the initial design had not addressed.

Key Findings

1. The framework came before the tool. The institution built a Human-Centered AI Assessment Framework around three principles — human judgment as non-negotiable interpretive authority, ethical stewardship of AI-augmented evidence, and formative human-AI partnership — so that institutional values rather than platform capabilities would determine how AI participated in assessment.
2. The problem was continuity, compliance, and consistency, not technology. The college had an established assessment infrastructure aligned with Higher Learning Commission criteria, but participation and documentation quality varied across programs; staggered deadlines, faculty turnover, and decentralized documentation weakened continuity, and rubric interpretation and artifact selection were inconsistent.
3. Faculty endorsement was strong and near-unanimous. Mean ratings were four or higher on every closed-ended item; all ten participants selected the highest option for future-use intent (Q15: M = 5.00, SD = 0.00), and the highest mean concerned thinking more carefully about evidence of student learning (Q9: M = 4.67, SD = 0.50).
4. Validation was reported but not universal. Six of the ten participants reported editing, correcting, adding context to, or rejecting AI-generated output consistently or sometimes, while four reported using the output as generated; the survey could not distinguish between output that needed no correction and output accepted without review.
5. Fluent output encouraged trust before verification. One participant reported that fabricated institutional details and inaccurate course classifications were dangerous precisely because the output was impressive at first: "The dangerous thing is that I was so impressed at first that I was skimming and not reading."
6. A platform failure migrated work to an unapproved tool. After roughly six hours of document-processing failures and hallucinated content in a separate program-review assistant, the participant moved the task to a personal ChatGPT account and completed it in about two minutes, then clarified that the future-use rating of five referred to ChatGPT rather than the institutional platform.
7. Data governance and support emerged as first-order concerns. Participants raised uncertainty about retention and use of institutional information submitted to external tools, requested more prompt training and continued practice, and asked whether the Wizard would retain updated assessment information — needs the institution met with revised instructions and a structured support protocol.

A framework built from institutional values

The pilot's starting point was a magisterial frame rather than a vendor: Pope Leo XIV's Apostolic Letter Drawing New Maps of Hope, published in October 2025, names AI literacy as one of only three new priorities added to the Global Compact on Education and calls institutions to "educate in a judicious use of technology and of AI, placing the person before the algorithm." The authors treat that as a design principle, not background, and translate Catholic social teaching into three operational principles. Human judgment as non-negotiable interpretive authority means AI may organize evidence, identify preliminary patterns, generate questions, and draft documentation, but may not determine what students have learned, establish final findings, or make curricular decisions. Ethical stewardship of AI-augmented evidence is operationalized through de-identification requirements, explicit faculty review of generated content, and the expectation that final findings remain human-led. Formative human-AI partnership draws on subsidiarity: technology should support faculty in carrying out their responsibilities rather than assume them. The institution's own problem was not technological but structural and cultural — continuity, compliance, and consistency across a program-assessment cycle that had to serve Higher Education and five graduate degree programs with limited assessment staffing.

The pilot: Education Day and the Assessment Wizard

The intervention had two parts. The Assessment Wizard is a GenAI-supported conversational process scaffold aligned with Higher Learning Commission criteria; it is not a grading tool, does not evaluate individual student work, and is not student-facing or autonomous. It orients the faculty member to the process, confirms the program outcomes under review, identifies evidence already collected, and then guides evidence submission one question at a time — describing the evidence, identifying observed patterns, explaining contextual factors, and considering improvement actions — with outputs framed as observations requiring verification rather than final findings. Education Day, held on May 6, 2026, opened with a half-day workshop on responsible GenAI engagement that introduced the CRAFT prompting framework (Context, Role, Action, Format, Tone), then moved into supervised practice in which faculty worked individually and in small groups with de-identified evidence they had brought, supported by assessment leadership, technical assistance, and a consultant. Faculty had also received earlier exposure during a Fall 2025 Faculty Welcome Week session on designing curricula for the AI era. The study did not measure participants' prior AI knowledge, so the authors treat these activities as shared preparation rather than evidence that all faculty began with equivalent literacy.

What the survey and the reflections showed

The design was a bounded case study with a convergent mixed-methods approach: a 16-item post-Education Day survey administered through Microsoft Forms, open-ended reflections embedded in the survey, and institutional process artifacts used to contextualize the case. Closed-ended results were uniformly positive, with means of four or higher on every item; the highest was future-use intent (M = 5.00, SD = 0.00), followed by thinking more carefully about evidence of student learning (M = 4.67, SD = 0.50) and retaining control over interpretation (M = 4.56, SD = 0.53). Variation was greatest on transparency about AI use (SD = 1.45) and on whether professional judgment rather than AI was responsible for final decisions (SD = 1.41). The behavioral item split the cohort: six of ten participants reported editing, correcting, adding context to, or rejecting AI-generated output consistently or sometimes, while four reported using the output as generated. Thematic analysis of nine accounts produced five themes — efficiency and cognitive scaffolding, human-in-the-loop validation, platform reliability and trust, data governance and ethical stewardship, and institutional support infrastructure — and the authors retained one participant's account of substantial platform difficulty as a negative case rather than discarding it.

The enthusiasm–vigilance paradox and governance

The authors name a tension they call the enthusiasm–vigilance paradox: the qualities that make generative AI attractive — speed, fluency, responsiveness, apparent confidence — may also make users less likely to scrutinize its output. The negative case sharpens it. One participant spent approximately six hours attempting a Faculty Qualifications section in a separate program-review assistant, encountering document-processing failures and hallucinated content such as fabricated degree-granting institutions, invented mission pillars, and inaccurate course categories; the same task took approximately two minutes in a personal ChatGPT account. The authors are careful about attribution: the incident occurred in the program-review assistant, not in the Assessment Wizard, which nonetheless produced smaller inaccuracies such as an incorrect course title and an incorrect capstone designation. The governance lesson is that a support protocol alone is insufficient. Responsible use cannot depend on each faculty member interpreting vendor terms, troubleshooting failures, or deciding which institutional information may be submitted; the institution needs to identify approved platforms by information category, clarify what may be uploaded, provide an accessible escalation path, assign responsibility for vendor review, and maintain the Wizard's knowledge base. The authors' formulation is that the safest process must also be the easiest process to follow, or technical friction will keep pushing users toward external alternatives.

What this means for practice

• Assessment professionals. Put formation before implementation: begin each assessment cycle with a structured orientation that connects institutional mission, the assessment cycle, AI literacy, data responsibilities, and the limits of the technology.
• Assessment professionals. Embed verification checkpoints in the workflow — require faculty to compare AI-generated summaries with source materials and confirm outcomes, course information, quantitative results, institutional claims, and proposed actions before content enters the official record.
• Teaching Treat fluent output as a reason to read more carefully, not less: the participant who reported fabricated institutional details described being "so impressed at first that I was skimming and not reading."
• Faculty developers. Build practice in prompting, verification, data handling, and responsible use, and curate the tool's knowledge base each cycle so program outcomes, curriculum maps, and terminology stay current.
• Administrators Make the approved pathway the easiest one: publish data-classification guidance, name a contact, define help-ticket response times, and assign an owner and follow-up measure to each improvement action.

Limitations

• The study is an exploratory account of a single bounded pilot with 10 participants — the complete population of program directors and department chairs at one small Catholic liberal arts college — so it precludes statistical inference and limits transferability.
• One participant did not provide open-ended responses, so the qualitative analysis reflects nine accounts, and item-level valid response counts varied across the closed-ended items.
• The lead researcher designed the intervention and was the primary analyst, which the authors acknowledge creates the possibility of Bias Mitigation despite collaborative review by the Director of Assessment and an external consultant.
• Data were collected immediately after Education Day, so the findings capture initial perceptions, reported behaviors, and intentions rather than sustained use, report quality, implemented improvements, or effects on student learning.
• The most substantial reliability incident occurred in a separate program-review assistant rather than the Assessment Wizard, and the unanimous future-use result partly reflects one participant's clarification that the rating of five referred to ChatGPT rather than the institutional platform.
• Participants' prior AI knowledge and experience were not measured, so differences in familiarity may have influenced confidence, validation behavior, platform preferences, and willingness to use AI again.

Citation

Ravesloot, H., Copeland, A., & Pryor, A. (2026). From Chaos to Consistency: A Human-Centered Framework for AI-Supported Assessment at a Mission-Driven Liberal Arts College. Intersection: A Journal at the Intersection of Assessment and Learning, 7(5), 56–91.