all_agents_tutorials/essay_grading_system_langgraph.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Essay Grading System using LangGraph\n",
    "\n",
    "## Overview\n",
    "This notebook presents an automated essay grading system implemented using LangGraph and an LLM model. The system evaluates essays based on four key criteria: relevance, grammar, structure, and depth of analysis.\n",
    "\n",
    "## Motivation\n",
    "Automated essay grading systems can significantly streamline the assessment process in educational settings, providing consistent and objective evaluations. This implementation aims to demonstrate how large language models and graph-based workflows can be combined to create a sophisticated grading system.\n",
    "\n",
    "## Key Components\n",
    "1. State Graph: Defines the workflow of the grading process\n",
    "2. LLM Model: Provides the underlying language understanding and analysis\n",
    "3. Grading Functions: Separate functions for each evaluation criterion\n",
    "4. Conditional Logic: Determines the flow of the grading process based on interim scores\n",
    "\n",
    "## Method\n",
    "The system follows a step-by-step approach to grade essays:\n",
    "\n",
    "1. Content Relevance: Assesses how well the essay addresses the given topic\n",
    "2. Grammar Check: Evaluates the essay's language usage and grammatical correctness\n",
    "3. Structure Analysis: Examines the organization and flow of ideas in the essay\n",
    "4. Depth of Analysis: Gauges the level of critical thinking and insight presented\n",
    "\n",
    "Each step is conditionally executed based on the scores from previous steps, allowing for early termination of low-quality essays. The final score is a weighted average of all individual component scores.\n",
    "\n",
    "## Conclusion\n",
    "This notebook demonstrates a flexible and extensible approach to automated essay grading. By leveraging the power of large language models and a graph-based workflow, it offers a nuanced evaluation of essays that considers multiple aspects of writing quality. This system could be further refined and adapted for various educational contexts, potentially improving the efficiency and consistency of essay assessments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## System Workflow\n",
    "\n",
    "<div style=\"text-align: center;\">\n",
    "\n",
    "<img src=\"../images/essay_grading_system_langgraph.svg\" alt=\"essay grading system langgraph\" style=\"width:80%; height:auto;\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and Imports\n",
    "\n",
    "This cell imports necessary libraries and sets up the OpenAI API key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import TypedDict\n",
    "from langgraph.graph import StateGraph, END\n",
    "from langchain_openai import ChatOpenAI\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "import re\n",
    "\n",
    "# Load environment variables and set OpenAI API key\n",
    "load_dotenv()\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_API_KEY')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## State Definition\n",
    "\n",
    "This cell defines the State class, which represents the state of our grading process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class State(TypedDict):\n",
    "    \"\"\"Represents the state of the essay grading process.\"\"\"\n",
    "    essay: str\n",
    "    relevance_score: float\n",
    "    grammar_score: float\n",
    "    structure_score: float\n",
    "    depth_score: float\n",
    "    final_score: float"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language Model Initialization\n",
    "\n",
    "This cell initializes the ChatOpenAI model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the ChatOpenAI model\n",
    "llm = ChatOpenAI(model=\"gpt-4o-mini\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grading Functions\n",
    "\n",
    "This cell defines the functions used in the grading process, including score extraction and individual grading components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_score(content: str) -> float:\n",
    "    \"\"\"Extract the numeric score from the LLM's response.\"\"\"\n",
    "    match = re.search(r'Score:\\s*(\\d+(\\.\\d+)?)', content)\n",
    "    if match:\n",
    "        return float(match.group(1))\n",
    "    raise ValueError(f\"Could not extract score from: {content}\")\n",
    "\n",
    "def check_relevance(state: State) -> State:\n",
    "    \"\"\"Check the relevance of the essay.\"\"\"\n",
    "    prompt = ChatPromptTemplate.from_template(\n",
    "        \"Analyze the relevance of the following essay to the given topic. \"\n",
    "        \"Provide a relevance score between 0 and 1. \"\n",
    "        \"Your response should start with 'Score: ' followed by the numeric score, \"\n",
    "        \"then provide your explanation.\\n\\nEssay: {essay}\"\n",
    "    )\n",
    "    result = llm.invoke(prompt.format(essay=state[\"essay\"]))\n",
    "    try:\n",
    "        state[\"relevance_score\"] = extract_score(result.content)\n",
    "    except ValueError as e:\n",
    "        print(f\"Error in check_relevance: {e}\")\n",
    "        state[\"relevance_score\"] = 0.0\n",
    "    return state\n",
    "\n",
    "def check_grammar(state: State) -> State:\n",
    "    \"\"\"Check the grammar of the essay.\"\"\"\n",
    "    prompt = ChatPromptTemplate.from_template(\n",
    "        \"Analyze the grammar and language usage in the following essay. \"\n",
    "        \"Provide a grammar score between 0 and 1. \"\n",
    "        \"Your response should start with 'Score: ' followed by the numeric score, \"\n",
    "        \"then provide your explanation.\\n\\nEssay: {essay}\"\n",
    "    )\n",
    "    result = llm.invoke(prompt.format(essay=state[\"essay\"]))\n",
    "    try:\n",
    "        state[\"grammar_score\"] = extract_score(result.content)\n",
    "    except ValueError as e:\n",
    "        print(f\"Error in check_grammar: {e}\")\n",
    "        state[\"grammar_score\"] = 0.0\n",
    "    return state\n",
    "\n",
    "def analyze_structure(state: State) -> State:\n",
    "    \"\"\"Analyze the structure of the essay.\"\"\"\n",
    "    prompt = ChatPromptTemplate.from_template(\n",
    "        \"Analyze the structure of the following essay. \"\n",
    "        \"Provide a structure score between 0 and 1. \"\n",
    "        \"Your response should start with 'Score: ' followed by the numeric score, \"\n",
    "        \"then provide your explanation.\\n\\nEssay: {essay}\"\n",
    "    )\n",
    "    result = llm.invoke(prompt.format(essay=state[\"essay\"]))\n",
    "    try:\n",
    "        state[\"structure_score\"] = extract_score(result.content)\n",
    "    except ValueError as e:\n",
    "        print(f\"Error in analyze_structure: {e}\")\n",
    "        state[\"structure_score\"] = 0.0\n",
    "    return state\n",
    "\n",
    "def evaluate_depth(state: State) -> State:\n",
    "    \"\"\"Evaluate the depth of analysis in the essay.\"\"\"\n",
    "    prompt = ChatPromptTemplate.from_template(\n",
    "        \"Evaluate the depth of analysis in the following essay. \"\n",
    "        \"Provide a depth score between 0 and 1. \"\n",
    "        \"Your response should start with 'Score: ' followed by the numeric score, \"\n",
    "        \"then provide your explanation.\\n\\nEssay: {essay}\"\n",
    "    )\n",
    "    result = llm.invoke(prompt.format(essay=state[\"essay\"]))\n",
    "    try:\n",
    "        state[\"depth_score\"] = extract_score(result.content)\n",
    "    except ValueError as e:\n",
    "        print(f\"Error in evaluate_depth: {e}\")\n",
    "        state[\"depth_score\"] = 0.0\n",
    "    return state\n",
    "\n",
    "def calculate_final_score(state: State) -> State:\n",
    "    \"\"\"Calculate the final score based on individual component scores.\"\"\"\n",
    "    state[\"final_score\"] = (\n",
    "        state[\"relevance_score\"] * 0.3 +\n",
    "        state[\"grammar_score\"] * 0.2 +\n",
    "        state[\"structure_score\"] * 0.2 +\n",
    "        state[\"depth_score\"] * 0.3\n",
    "    )\n",
    "    return state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workflow Definition\n",
    "\n",
    "This cell defines the grading workflow using StateGraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the StateGraph\n",
    "workflow = StateGraph(State)\n",
    "\n",
    "# Add nodes to the graph\n",
    "workflow.add_node(\"check_relevance\", check_relevance)\n",
    "workflow.add_node(\"check_grammar\", check_grammar)\n",
    "workflow.add_node(\"analyze_structure\", analyze_structure)\n",
    "workflow.add_node(\"evaluate_depth\", evaluate_depth)\n",
    "workflow.add_node(\"calculate_final_score\", calculate_final_score)\n",
    "\n",
    "# Define and add conditional edges\n",
    "workflow.add_conditional_edges(\n",
    "    \"check_relevance\",\n",
    "    lambda x: \"check_grammar\" if x[\"relevance_score\"] > 0.5 else \"calculate_final_score\"\n",
    ")\n",
    "workflow.add_conditional_edges(\n",
    "    \"check_grammar\",\n",
    "    lambda x: \"analyze_structure\" if x[\"grammar_score\"] > 0.6 else \"calculate_final_score\"\n",
    ")\n",
    "workflow.add_conditional_edges(\n",
    "    \"analyze_structure\",\n",
    "    lambda x: \"evaluate_depth\" if x[\"structure_score\"] > 0.7 else \"calculate_final_score\"\n",
    ")\n",
    "workflow.add_conditional_edges(\n",
    "    \"evaluate_depth\",\n",
    "    lambda x: \"calculate_final_score\"\n",
    ")\n",
    "\n",
    "# Set the entry point\n",
    "workflow.set_entry_point(\"check_relevance\")\n",
    "\n",
    "# Set the exit point\n",
    "workflow.add_edge(\"calculate_final_score\", END)\n",
    "\n",
    "# Compile the graph\n",
    "app = workflow.compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Essay Grading Function\n",
    "\n",
    "This cell defines the main function to grade an essay using the defined workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def grade_essay(essay: str) -> dict:\n",
    "    \"\"\"Grade the given essay using the defined workflow.\"\"\"\n",
    "    initial_state = State(\n",
    "        essay=essay,\n",
    "        relevance_score=0.0,\n",
    "        grammar_score=0.0,\n",
    "        structure_score=0.0,\n",
    "        depth_score=0.0,\n",
    "        final_score=0.0\n",
    "    )\n",
    "    result = app.invoke(initial_state)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Essay\n",
    "\n",
    "This cell contains a sample essay for testing the grading system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_essay = \"\"\"\n",
    "    The Impact of Artificial Intelligence on Modern Society\n",
    "\n",
    "    Artificial Intelligence (AI) has become an integral part of our daily lives, \n",
    "    revolutionizing various sectors including healthcare, finance, and transportation. \n",
    "    This essay explores the profound effects of AI on modern society, discussing both \n",
    "    its benefits and potential challenges.\n",
    "\n",
    "    One of the most significant impacts of AI is in the healthcare industry. \n",
    "    AI-powered diagnostic tools can analyze medical images with high accuracy, \n",
    "    often surpassing human capabilities. This leads to earlier detection of diseases \n",
    "    and more effective treatment plans. Moreover, AI algorithms can process vast \n",
    "    amounts of medical data to identify patterns and insights that might escape \n",
    "    human observation, potentially leading to breakthroughs in drug discovery and \n",
    "    personalized medicine.\n",
    "\n",
    "    In the financial sector, AI has transformed the way transactions are processed \n",
    "    and monitored. Machine learning algorithms can detect fraudulent activities in \n",
    "    real-time, enhancing security for consumers and institutions alike. Robo-advisors \n",
    "    use AI to provide personalized investment advice, democratizing access to \n",
    "    financial planning services.\n",
    "\n",
    "    The transportation industry is another area where AI is making significant strides. \n",
    "    Self-driving cars, powered by complex AI systems, promise to reduce accidents \n",
    "    caused by human error and provide mobility solutions for those unable to drive. \n",
    "    In logistics, AI optimizes routing and inventory management, leading to more \n",
    "    efficient supply chains and reduced environmental impact.\n",
    "\n",
    "    However, the rapid advancement of AI also presents challenges. There are concerns \n",
    "    about job displacement as AI systems become capable of performing tasks \n",
    "    traditionally done by humans. This raises questions about the need for retraining \n",
    "    and reskilling the workforce to adapt to an AI-driven economy.\n",
    "\n",
    "    Privacy and ethical concerns also arise with the increasing use of AI. The vast \n",
    "    amount of data required to train AI systems raises questions about data privacy \n",
    "    and consent. Additionally, there are ongoing debates about the potential biases \n",
    "    in AI algorithms and the need for transparent and accountable AI systems.\n",
    "\n",
    "    In conclusion, while AI offers tremendous benefits and has the potential to solve \n",
    "    some of humanity's most pressing challenges, it also requires careful consideration \n",
    "    of its societal implications. As we continue to integrate AI into various aspects \n",
    "    of our lives, it is crucial to strike a balance between technological advancement \n",
    "    and ethical considerations, ensuring that the benefits of AI are distributed \n",
    "    equitably across society.\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grading the Sample Essay\n",
    "\n",
    "This cell demonstrates how to use the grading system on the sample essay and display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Essay Score: 0.86\n",
      "\n",
      "Relevance Score: 1.00\n",
      "Grammar Score: 0.90\n",
      "Structure Score: 0.85\n",
      "Depth Score: 0.70\n"
     ]
    }
   ],
   "source": [
    "# Grade the sample essay\n",
    "result = grade_essay(sample_essay)\n",
    "\n",
    "# Display the results\n",
    "print(f\"Final Essay Score: {result['final_score']:.2f}\\n\")\n",
    "print(f\"Relevance Score: {result['relevance_score']:.2f}\")\n",
    "print(f\"Grammar Score: {result['grammar_score']:.2f}\")\n",
    "print(f\"Structure Score: {result['structure_score']:.2f}\")\n",
    "print(f\"Depth Score: {result['depth_score']:.2f}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}