Skip to main content

VMChecker TA Guide for Assignments

1. Introduction​

VMchecker next is a new type of homework testing environment. It makes it so a student and a teacher can both use the same infrastructure in order to test homework with minimal input.

When you want to add a new assignment to VMchecker next you will need to have the following components ready:

  • a private assignment repository, based on which the homework will be tested. This repository will have to remain private, as assignments are tested by creating a new branch for each submission through Moodle;
  • a public repository that is a mirror of the private repository which can be forked privately by students in order to test their homework using the same scripts and containers as on Moodle;
  • a Moodle assignment where students will have to upload their homework;
  • a Moodle VMchecker block where TAs can configure the plugin that controls homework checking.

2. Gitlab Setup​

The private and public repositories must have the following structure:

├── checker/
│   ├── checker.sh
│   ├── input/
│   │   ├── test0
│   └── output/
│       ├── test0.ref
├── Dockerfile
├── .gitlab-ci.yml
├── local.sh
└── src/
    └── skel.c

It is recommended to fork https://gitlab.cs.pub.ro/vmchecker/vmchecker-next-assignment for a faster start.

If you are working on Gitlab.com you can import the repository by going from the main page New Project > Import Project > Repository by URL and then filling in the Git repository URL with: https://gitlab.cs.pub.ro/vmchecker/vmchecker-next-assignment.git.

The project also contains the solution under src/solution.c if you want to test your GitLab setup before porting your assignment.

A typical assignment will have the above structure with the following files and folders of interest to teaching assistants:

  • src/ - Here the student will develop their solution according to the assignment's specification. Here, you can add a skeleton for the assignment.
  • local.sh - A utility script used to check the assignment locally. It simulates a Gitlab pipeline.
  • Dockerfile - Used to build the docker image for the Gitlab pipeline. The following is an example that should satisfy most cases:
FROM jokeswar/base-ctl
COPY ./checker ${CHECKER_DATA_DIRECTORY}

NOTE: The base-ctl image is based on the ubuntu:20.04 image and has the following additional packages installed: build-essential and python3.

NOTE: When the image is built the entire checker is copied into the docker image, to prevent the user from changing tests.

  • .gitlab-ci.yml - The definition of the pipeline. You must change the image used to the one you built;
  • checker/ - The checker directory;
  • checker/checker.sh - The entry point of the checker. When execution is passed to this script, the current working directory is set to the checker/ folder.

NOTE: In order for VMChecker Next to be able to grade the assignment it will look for the last string that matches the following regex: Total:\ *([0-9]+) (the grade must be an integer).

Valid format:

              Total:100/100
              Total:    0/100
Total:50/100

Invalid format:

              Total: 55.37/100       # The grade will be set as 55, not 55.37 on Moodle

NOTE: You will need to generate a project access token that will allow VMChecker to access Gitlab's API. You will need to give the token access to api, read_api, read_registry, write_registry, read_repository, write_repository and the role should be at least Developer. Also, the project ID for the private repository is needed.

First Changes Needed​

  1. Change the DEFAULT_IMAGE_NAME with the name of the public repository of your homework in the format groupname/projectname. This variable is found in the local.sh script
  2. Change the image used to the same name set as above in the .gitlab-ci.yml file on line 29

2.1. Disable Merge Requests for the Public Repository​

To avoid the possibility of students accidentally sharing their solution, Merge Requests for the public repository must be disabled. To do that:

  1. Go to your public project page
  2. On the left panel under Settings select General
  3. Under the Visibility, project features, permissions disable Merge Requests.

mr_disable

2.2. Set up a CD Pipeline for the Checker Image​

In .gitlab-ci.yml set the image name for the checker step to registry.gitlab.com/\<group-name>/\<project-name>:\<tag> (e.g: registry.gitlab.com/ASC/Assignment-3-Public:latest).

2.2.1. Build the Checker Image Using Gitlab Pipelines​

Set up the BUILD_DOCKER_IMAGE pipeline environment variable to true for your public repository. Each time a new commit is made to the repository a new Docker image build will be triggered. The image will be uploaded to Gitlab's own Docker registry.

2.2.2. Build the Checker Image Manually​

./local.sh docker build
./local.sh docker push --user "$GITLAB_USERNAME" --token "$GITLAB_ACCESS_TOKEN"

To create a GITLAB_ACCESS_TOKEN access token follow the documentation on Gitlab.

For more information on local.sh run ./local.sh -h.

2.3. Find the Project ID​

  1. Go to your project page
  2. The ID is found under the project name

project_id

2.4. Setup the Gitlab Runner​

Lastly, we need to register our Gitlab Runner, so that our assignment verification jobs get checked. To do that we go to our project and click on Settings > CI/CD > Project Runners. While we are here make sure that the option Enable shared runners for this project is disabled!

add-runner

Then you will have to select your Operating System and whether to use Docker or Kubernetes to host the runner. Currently, we will use the Docker engine to start and host the GitLab Runner. In this step, make sure that Run untagged jobs is checked.

runner-config-1

The Docker runner container was already started as part of the docker-compose command. Finally, click on Create runner found at the bottom of the page

Now you will be presented with some instructions on how to register the runner instance on your machine with Gitlab. The official documentation is found here. In our case, the command to register the runner is:

docker exec -it vmchecker-test-runner-1 gitlab-runner register

runner-config-2

3. Moodle Setup​

First, set up a Moodle assignment with the following additional settings:

  • Submission types → Submission types → File submissions
  • Submission types → Maximum number of uploaded files → 1
  • Submission types → Maximum submission size → 50kb (The maximum submission size of VMChecker Next is 10 MB)
  • Submission types → Accepted file types → zip
  • Feedback types → Offline grading worksheet → true (enables feedback download)

Next, add a new VMChecker block on the current page by clicking on Add a block and selecting VMChecker block from the list.

add_block

Click on the gear and select Configure VMChecker block. Here you must fill in a couple of options for the VMChecker settings header:

  • Enable auto-grading of submissions - If you want the assignment to be automatically graded with the mark from the checker
  • Gitlab repository ID - The project ID of your private repository
  • Gitlab private token - The previously generated private access token
  • Gitlab branch - The parent branch name from which student branches will have their changes merged and tested in the private repository
  • Assignment - From the list select the assignment you want VMChecker to manage

WARNING: There is a one-to-one relationship between an assignment and a VMChecker block. Make sure there is only one VMChecker block per assignment.

Save!

4. Test Your Assignment​

After the setup is done, switch to the student test account and upload your submission. After a couple of minutes, your submission will run on the GitLab CI. You can check your private repository. When the pipeline is finished the trace will be set as feedback for your assignment. If auto-grading is enabled, your submission will also be marked.

NOTE: On Moodle, the feedback string will be computed from the trace found on the Gitlab pipeline and will be the first 300 lines of the content between the start of the checker string mark (i.e. <VMCK_NEXT_BEGIN>) and the end string mark (i.e. <VMCK_NEXT_END>).

<Pipeline output>
<VMCK_NEXT_BEGIN>
Test 1....0              ---|
.                           |
.                           |
.                           |   This will be the feedback for the student assignment.
Test n...0                  |
                            |
    Total 0/100          ---|
<VMCK_NEXT_END>
<some more Pipeline output>

5. Grading an Assignment​

5.1. Download All Submissions​

  1. Go to the course page on Moodle
  2. Click on the assignment
  3. Click on the gear and from the drop-down select Download all submissions

download_all

NOTE: If you do not have permission, the assignment page will show the upload form only.

5.2. Download Grading Worksheet​

  1. Go to the course page on Moodle
  2. Click on the assignment
  3. Click on View all submissions
  4. From the grading action select Download grading worksheet

download_worksheet

NOTE: You can grade submissions and give feedback in the downloaded CSV.

5.3. Check a Submission on Gitlab​

To check the pipeline of a certain student, go to the assignment's private repository on GitLab. From the menu on the left select Repository → Branches. Every student submission has a separate branch of the following format ${username}-${year}-${month}-${day}-${hour}-${minute}-${second}-${uuid}. Using the search bar, look for the desired submission.

branches

To check the user code click on the branch name and check the src folder.

If you want to see the pipeline output, click on the pipeline status icon.

pipeline_select

Next, click on Jobs and select the first job ID.

jobs

5.4. Import Grades​

  1. Go to the course page on Moodle
  2. Click on the assignment
  3. Click on View all submissions
  4. From the grading action select Upload grading worksheet

upload_worksheet

  1. Upload the worksheet downloaded previously
  2. After you preview the changes click Confirm

6. VMChecker block​

6.1. Recheck Submission for ...​

In the VMChecker block for the desired assignment select Recheck submission for ..., as action. Search in the list for the student, you can use the search bar for a faster lookup. Finally press Run action.

NOTE: Only students who have submitted their assignments will be found on the list.

recheck_for

6.2. Recheck All​

In the VMChecker block for the desired assignment select Recheck all submissions, as action, and press Run action. All students who have submitted their homework will have their last submission rechecked.

recheck_all