Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add detection for mismatch in Autograder results and assessment problems #2112

Merged
merged 4 commits into from
Mar 11, 2024
Merged

Add detection for mismatch in Autograder results and assessment problems #2112

merged 4 commits into from
Mar 11, 2024

Conversation

evanyeyeye
Copy link
Member

@evanyeyeye evanyeyeye commented Feb 26, 2024 •
edited
Loading

Description

  • Added mismatch notification on submission views.
  • Improvement and fix to existing mismatch detection for autograding feedback.

image
image

Motivation and Context

Closes #2087.

How Has This Been Tested?

  • Tested submission with all problems mismatched.
  • Tested submission with some problems mismatched.
  • Tested submission with no problems mismatched.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

@evanyeyeye
Implement mismatch notification

Verified

This commit was signed with the committer’s verified signature.
cg2v Chaskiel Grundman
GPG key ID: 0148619DCF2274B3
Verified
Learn about vigilant mode
d61858b
@evanyeyeye
Add error message to feedback
72a5ed4
@coderabbitai coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 26, 2024 •
edited
Loading

Walkthrough

Walkthrough

The changes address the issue of mismatches between Autograder results and assessment problems by introducing functionality to detect and report missing problems. This includes recording missing problems during autograding, updating the database schema to support this data, and adjusting the UI to notify users of any discrepancies. The adjustments ensure that missing problems are clearly identified and communicated, improving the transparency and usability of the autograding feedback process.

Changes

File Path Changes
app/helpers/.../assessment_autograde_core.rb Introduced handling for missing problems during autograding and adjusted logic for setting problem scores.
app/views/assessments/.../_submission_history_row.html.erb Added UI logic to display a message for missing problems in autograder results.
db/migrate/.../20240226194217_add_missing_problems_to_submissions.rb Added a migration to include a missing_problems text column in the submissions table.
db/schema.rb Updated schema to reflect the addition of the missing_problems column in the submissions table.

Assessment against linked issues

Objective Addressed Explanation
Better detection of mismatch in Autograder results and assessment problems (#2087) ✅
Notification for instructors about missing problems in Autograder results (#2087) ✅

The changes effectively address the objectives outlined in issue #2087 by implementing mechanisms for better detection and reporting of mismatches between Autograder results and assessment problems. The introduction of a missing_problems field in the database, along with the UI enhancements for notifying instructors, directly tackles the problem of unreported mismatches, fulfilling the requirements of the linked issue without any apparent omissions.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit-tests for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit tests for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit tests.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

@damianhxy damianhxy requested review from a team and 20wildmanj and removed request for a team February 26, 2024 20:56
20wildmanj
20wildmanj approved these changes Mar 8, 2024
View reviewed changes
Copy link
Contributor

@20wildmanj 20wildmanj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

functionally looks good to me, just a style wise nit

@evanyeyeye evanyeyeye added this pull request to the merge queue Mar 11, 2024
Merged via the queue into master with commit 521d4ae Mar 11, 2024
5 checks passed
@evanyeyeye evanyeyeye deleted the problem-mismatch branch March 11, 2024 19:26
NicholasMy pushed a commit to UB-CSE-IT/Autolab that referenced this pull request Jan 10, 2025
@evanyeyeye @NicholasMy
Add detection for mismatch in Autograder results and assessment probl…
503d005 
…ems (autolab#2112)

* Implement mismatch notification

* Add error message to feedback

* Address nits

(cherry picked from commit 521d4ae)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@coderabbitai coderabbitai[bot]

@20wildmanj 20wildmanj

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Better detection of mismatch in Autograder results and assessment problems
2 participants
@evanyeyeye @20wildmanj