LLM-as-a-Grader: Automated Grading of Written Work with GPT

TL;DR

LLM-as-a-Grader is a study of how well GPT-4o grades student work: short answers to questions and project reports. The authors compared the model's grades with those of human instructors on a real computational linguistics course: 258 quiz answers and 14 project reports.

Main insight: GPT-4o shows a 0.98 correlation with human grades and matches exactly in 55% of cases. But there's a systematic skew: the model grades lower in 39% of cases and higher in only 6%. GPT is stricter about quantitative results and depth of analysis, while humans pay more attention to formatting, clarity and the quality of conclusions. On objective criteria (related work, approach) the grades are nearly identical. The gaps show up in the "soft" areas: style, narrative, rhetoric.

In practice: The researchers released a ready-made prompt and toolkit. The key to the method is a gold answer (reference) + a fixed set of scores + a requirement to explain any deduction. The model doesn't just assign a grade; it shows what it took points off for.

Method Blueprint

QUIZZES (short answers):
Question + Reference answer + Student answer → GPT → Score + Explanation (if below max)

REPORTS (long documents):
PDF → Text extraction → GPT grades each rubric section → Scores + Justification per section

Everything happens in one request. For reports, sections are graded separately but within a single prompt.

Example in Practice

Task: You're an English tutor. You're grading 30 essays on "Why reading books matters". You need to assign scores and give feedback.

Prompt:

You are grading a student's essay. Follow these rules:

1. The maximum score for this assignment is 5.
2. You can only choose from these scores: [0, 1, 2, 3, 4, 5].
3. Return the grade in the format: "Score: X" where X is one of the values above.
4. If you give less than 5, write a short explanation on the next line, starting with "Explanation:".
5. If you give 5, write nothing after "Score: 5".
6. If the essay addresses the topic and contains at least 2-3 key ideas, give the maximum. Even if there are minor flaws, as long as understanding of the topic is clear, give full marks.
7. If the essay is empty or completely off-topic, give 0.

Criteria for a good essay:
- It has a thesis (main idea)
- Arguments or examples are given
- There's a conclusion that ties back to the opening
- Logical structure

Topic: Why reading books matters

Student essay:
"""
Books are important because they develop your brain. When you read, you learn new words. Books also help you relax after school. My favorite author is Harry Potter. Anyway, reading is good for you.
"""

Score:

Result: The model will give a score (probably 3-4) and an explanation: it will note a basic grasp of the topic but no clear structure, shallow arguments, and that "Harry Potter is an author" is a factual error. The output format will be strictly: "Score: X" + "Explanation: ..."

Why It Works

LLMs are good at comparing texts by meaning. With a reference answer, the model can judge how well the student "hit" the key ideas. This isn't lexical matching (word overlap) but semantic alignment (overlap in meaning). The student can phrase things differently and the model will still get it.

A fixed set of scores removes arbitrariness. Instead of "rate from 0 to 100", we say "choose from [0, 0.1, 0.2]". The model doesn't invent a scale; it works within set boundaries. Less variability means more predictability.

Requiring an explanation for deductions makes grading transparent. The model can't quietly take points off; it has to give a reason. That helps the grader (you see the logic) and the student (they know what to fix).

Control levers:

• Score scale → narrow it for simple tasks (0/1), widen it for complex ones (0-10)
• Criteria in the prompt → add specific requirements ("must include a real-life example")
• "If understanding is clear, give full marks" → remove this line if you want strict grading
• Explanation on deductions → add "Explanation: even for full marks" to always get detailed feedback

Prompt Template

For short answers:

You are grading a student's answer. Follow these rules:

1. Maximum score: {max_score}.
2. Allowed scores: {score_list}.
3. Response format: "Score: X" where X is one of the values above.
4. If the score is below the maximum, add "Explanation:" with the reason on the next line.
5. If the score equals the maximum, add nothing after the score.
6. If the answer contains the key ideas of the reference, give the maximum, even if there are minor gaps.
7. If the answer is empty or off-topic, give 0.

Question: {question}
Reference answer: {reference}
Student answer: {answer}

Score:

For reports/documents:

You are grading a {document_type}. Score each section against the rubric.

Rubric:
{section_1}: up to {points_1} — {criteria_1}
{section_2}: up to {points_2} — {criteria_2}
...

For each section:
1. Write "Section: {name}"
2. "Score: X out of Y"
3. "Comment:" — 1-2 sentences on what's good or what's missing

At the end:
- Total: sum of scores
- Summary: 2-3 sentences on the work as a whole

Document:
"""
{document_text}
"""

Placeholders:

• {max_score}: the maximum score for the question (e.g., 5)
• {score_list}: the allowed values: [0, 1, 2, 3, 4, 5]
• {reference}: the correct answer or key ideas
• {document_type}: "essay", "report", "term paper"
• {section_N} / {criteria_N}: section names and what to assess in each

Limitations

⚠️ Conservatism: GPT systematically grades low: below the human in 39% of cases, above in only 6%. Factor this in when assigning final grades.

⚠️ Weak on "soft" criteria: Clarity, style, rhetoric, "how persuasive it is": this is where the model diverges from humans. These criteria need human review.

⚠️ A reference is required: The method works when there's a "gold answer" or a clear rubric. For subjective assignments ("write a creative essay") a reference is hard to build.

⚠️ Technical/open-ended sections: On the "Approach" and "Results" sections of reports, GPT diverged from humans the most. Where expert judgment of novelty or experiment quality is needed, the model is stricter.

How They Studied It

A team from Emory University used a real computational linguistics course: 50 students, 5 quizzes (258 answers), 14 project reports. Two graduate students graded the work independently, and their grades were then compared with GPT-4o's.

Quizzes used a 0–0.2 scale in steps of 0.1 (i.e., three possible scores). Reports used an 8-section rubric worth 9 points. The statistics were done properly: t-tests, Pearson correlations, and the Wilcoxon signed-rank test with Holm-Bonferroni correction for multiple comparisons.

What surprised them: the model isn't just "similar on average": a 0.98 correlation is very high. Yet there's a systematic downward shift. GPT is like a strict professor who rarely gives an A. Interestingly, the gaps are predictable: objective criteria (Related Work, format) are almost perfect, subjective ones (Results, Approach) show scatter.

Cost: the entire quiz set (14 questions × 50 students = 700 requests) cost less than $1 on GPT-4o.

Original from the Paper

You are grading a student's answer to a quiz question. Follow these rules:

1. The maximum score for this question is {full_score}.
2. You can only choose a score from this set: {valid_scores}.
3. Return the grade on a single line exactly in the format: "Grade: X" where X is one of {valid_scores}.
4. If you give a grade less than {full_score}, then on the next line provide a short single-sentence explanation starting with "Explanation:".
5. If you give a grade equal to {full_score}, do not provide any explanation or additional lines. End your response immediately after "Grade: {full_score}".
6. If the student's answer is relevant to the question and includes at least one or two key ideas from the gold answer, give the maximum score. Even if some details are missing, as long as the student shows general understanding, give the full points.

Example:
Question: All delimiters used in our implementation are punctuation marks. What types of tokens should not be split by such delimiters?
Gold Answer: URLs, email addresses, decimal numbers, and abbreviations
Student Answer: email and decimal numbers.
Grade: 0.2

7. If the student's answer is empty or obviously irrelevant, it cannot capture the essential idea, so you must give 0.

Question: {question}
Gold Answer: {gold_answer}
Student Answer: {student_answer}
Grade:

Context: This is the prompt for grading short quiz answers. The researchers used it for all 258 answers. Note the example: it's a few-shot demonstration showing that even an incomplete answer can earn full marks.

Adaptations and Extrapolations

💡 Adapting it for HR: Grading candidates' take-home assignments

You are grading a take-home assignment from a candidate for the {job_title} position. Follow these rules:

1. Maximum score: 10.
2. Allowed scores: [0, 2, 4, 6, 8, 10].
3. Format: "Score: X"
4. If the score is < 10, add "Explanation:" with the reason.
5. Grade against these criteria:
   - Understanding of the task (0-3)
   - Quality of the solution (0-4)
   - Presentation and communication (0-3)

Assignment: {assignment_description}
Reference solution or criteria: {criteria}
Candidate's solution:
"""
{solution}
"""

Score:

🔧 Technique: drop the "lenient" rule → stricter grading

Original rule:

If the student's answer is relevant to the question and includes at least one or two key ideas from the gold answer, give the maximum score.

Strict version:

Give the maximum score only if the answer covers ALL key ideas from the gold answer. Partial coverage = partial credit.

Effect: the model stops "forgiving" incomplete answers. Use it for exams where completeness matters.

🔧 Technique: add "Explanation: always" → detailed feedback

Replace rules 4-5 with:

4. After the grade, ALWAYS provide "Explanation:" with 1-2 sentences about what was good and what could be improved.

Effect: you get detailed feedback even on excellent work. Useful for teaching, not just grading.

Resources

Paper: "LLM-as-a-Grader: Practical Insights from Large Language Model for Short-Answer and Report Evaluation"

Authors: Grace Byun, Swati Rajwal, Jinho D. Choi, Emory University, Atlanta

Code and data: https://github.com/emorynlp/LLM-Grading