"AI-Assisted Decision Support for Defensible Transfer Report Assessment"

Assessing long student work requires lecturers to connect evidence across text, tables, and figures and justify judgments against shared criteria. This study uses design science research to develop and evaluate AI-assisted decision support for this task. General Business Administration Transfer Reports at the Swiss Institute of Business Administration provide the empirical case. Students use these Transfer Reports to apply academic concepts to practical business problems. The assignment brief specifies the expected work, while an analytical rubric defines the criteria and subcriteria, termed aspects, that guide its assessment. Lecturers are expected to assess each submitted Transfer Report against the assignment requirements and the rubric, assigning a score to each aspect. In routine practice, one lecturer assesses each Transfer Report. To examine whether different lecturers assess the same work consistently, five lecturers independently reassessed the same 30 anonymized Transfer Reports without AI assistance. Single-lecturer absolute agreement was low, ICC(2,1) = 0.330. The average gap between the highest and lowest score for the same Transfer Report was 17.7 points on a 60-point scale. The study combined findings from the human-only assessment study, relevant literature, and the institutional grading context to define 19 design requirements for an AI-assisted method that supports lecturers in assessing Transfer Reports. These requirements specify what the method should provide, including access to relevant evidence, traceable assessment proposals, and lecturer control over final decisions. To address these requirements, a four-stage prototype was developed iteratively by comparing alternative components and testing how they worked together. It prepares searchable text and table content and processes selected visual evidence. The prototype drafts provisional scores and reasons from the retrieved evidence. Selected reasons and feedback are then audited before the results are packaged with source references for lecturer review. Auditing and packaging leave scores unchanged. Evaluation of the outputs for the 30 target Transfer Reports included an internal review by the lecturer-researcher. The method generally proposed lower total scores than the original lecturer assessments and the average scores from the five independent re-ratings. Adding previously graded Transfer Reports as comparison material did not consistently reduce these differences. The standard version therefore does not use previously graded Transfer Reports as comparison material. The study connects an empirical grading diagnosis with explicit design requirements, an evidence-linked support method, and reusable evaluation procedures. A planned extension would report separately whether a Transfer Report meets mandatory submission conditions, such as the required length. Lecturers could then identify an unmet condition even when the report receives high scores for its content. So far, the method has been evaluated only for the Transfer Report assignment examined in this study. Further evaluation is needed to determine whether it can support other assignment types and whether lecturers using its evidence and suggestions assess the same work more consistently during actual grading.