Module 5: AI-Assisted Assessment Engineering

Assessment engineering begins by asking what evidence would demonstrate the intended learning outcome. The answer determines the task format more reliably than starting from “I need 30 questions.”

For each outcome, identify the performance and evidence needed. Recall may be sampled with selected-response items. Explanation may need short constructed responses. Complex reasoning may require worked solutions, essays, oral defense, design artifacts, presentations, or authentic performance tasks.

AI is most useful after this evidence decision. It can draft items, generate variations, propose rubrics, and audit coverage. It should not choose the construct solely because a format is easy to generate or grade.

A Table of Specifications (TOS) is an assessment blueprint linking content or outcomes with cognitive demand and item allocation. It makes weighting explicit before item generation.

A defensible TOS should record outcome/content, instructional emphasis, cognitive demand, item format, number of items, and percentage. Weighting can reflect contact time, curricular importance, prerequisite value, or institutional assessment policy—but the rationale should be stated. The final percentages and item counts must reconcile exactly with the planned assessment.

After AI drafts a TOS, inspect whether it overweights easy-to-test knowledge or underrepresents major outcomes. The blueprint is a control mechanism, not merely documentation after the exam has already been written.

Choose the format according to the evidence required. Selected response efficiently samples knowledge and interpretation when well constructed. Short answer supports concise recall or explanation. Essay supports sustained reasoning, synthesis, argument, or source use. Mathematical/technical problems reveal method, setup, calculation, units, and interpretation. Projects and presentations show integrated performance. Authentic performance tasks place decisions in realistic contexts.

A mixed assessment can balance breadth and depth. Avoid using AI to convert every outcome into MCQs. If the outcome requires learners to design, defend, demonstrate, or produce, the assessment should collect that kind of evidence.

A good single-best-answer MCQ has a focused stem, one defensible key, and plausible distractors representing real misconceptions. Avoid grammatical clues, option-length cues, unnecessary negatives, overlapping choices, trivial distractors, and “all/none of the above” unless a specific format requires them.

A useful AI workflow asks the model to produce the target outcome, intended cognitive level, stem, four options, key, rationale, and distractor logic for each item. Then audit the result separately. Try to defend each distractor. If two choices can reasonably be correct because the stem lacks a qualifier, repair the stem before release.

For technical or numerical MCQs, independently recompute the key and ensure distractors use compatible units and precision.

Constructed-response tasks need explicit scope. An essay prompt should identify the problem, expected evidence, source expectations, and criteria. A performance task should define the scenario, deliverable, constraints, audience, and what decisions must remain the learner's own.

When AI generates authentic tasks, check that “realistic” does not become unnecessarily complicated. The context should support the construct rather than add irrelevant reading burden.

For AI-permitted assessments, state what assistance is allowed and what evidence of process the learner must retain. Authenticity is strengthened by context-specific decisions, reflection, drafts, oral explanation, and traceable source use—not by relying on AI-content detectors.

Technical assessment items require more than a final numerical answer. The expected solution should expose method selection, governing relationship, setup, assumptions, intermediate checkpoints, units, and final interpretation.

Before releasing an AI-generated problem: solve it independently; verify all constants, data, and assumptions; confirm the problem is neither underdetermined nor internally contradictory; check that the requested precision is sensible; identify acceptable alternate methods; and create partial-credit checkpoints.

If the problem relies on a code, standard, or design provision, verify the current governing source rather than relying on model memory.

An analytic rubric separates criteria such as conceptual correctness, evidence, reasoning, technical execution, communication, or design quality. A holistic rubric produces one overall judgment. Analytic rubrics are usually easier to audit when AI is assisting with scoring because each criterion can be linked to specific evidence.

Performance levels should describe observable differences. Avoid vague labels such as “excellent” without stating what excellent performance looks like. Avoid criteria that unintentionally reward length, expensive production, or stylistic preferences unrelated to the outcome.

Before grading, test the rubric against a few representative responses. If two reviewers interpret a criterion differently, refine the language before scaling the workflow.

Parallel forms should measure the same intended outcomes at comparable cognitive demand and workload. Changing numbers or surface contexts does not automatically create equivalence. Compare forms on outcome/TOS coverage, cognitive demand, reading complexity, calculation burden, number of steps, likely completion time, and source/tool requirements.

Assessment validation also includes reviewing ambiguity, answer keys, fairness, accessibility, and alignment. A large item bank is useful only if each item remains defensible.

Use 90–110 minutes for the full module or 60–70 minutes for a condensed workshop because blueprinting and item review require hands-on time.

• Standards for Educational and Psychological Testing — AERA / APA / NCME. Professional standards for valid, fair, and appropriate educational assessment.
• Guidance for generative AI in education and research — UNESCO. Human-centred guidance for generative AI in education and research.
• Model optimization — OpenAI. Evaluation-first guidance for testing and improving AI outputs.