Markus Rubric

How To Set Up the Rubric on Markus

Log into Markus in the admin account (the account that you log in with your email). Go to the current assignment and click on the Marking Scheme tab.

Before adding any criteria, make sure that under the properties tab in the marking scheme section (Properties -> Marking Scheme), the “rubric” option is selected.

*Note that for assignment 0, midterms and finals this may be different.

There are two ways to set up the MarkUs rubric:

The first is to manually input the criteria and weights on MarkUs. Alternatively, you can create and edit the following on an Excel file and upload it as a .csv file to MarkUs. The latter will be covered at the end of this section.

Click “Add Criterion” to add a category on the rubric. Give it a meaningful name and a weight. Note that each rubric category has a maximum level of 4 by default, so the total marks that the category will be out of should be 4 times the weight. If you want the assignment to be marked out of 100 in total, make sure the sum of the weights of all the criteria is 25.0.

*** Please be careful not to include metacharacters in the criteria names since they complicate the job of the Autotesting team. A metacharacter is a character that has a special meaning to a program, in this case the scripts written by the Autotesting team. For example, in some cases (e.g. if a .csv rubric is uploaded), if the name "Q01a: Correctness" is copy-and-pasted into a terminal, it may contain a metacharacter (e.g. \377) which is invisible in MarkUs. Try to check with the Autotesting that this is not an issue after uploading a rubric onto MarkUs.

Note that you might find it helpful to split a criterion into two parts if you are spotchecking two different questions. For example, if you are spot-checking Q1 and Q5 for contract correctness, you can make two separate criteria titled "Q1: Contract Correctness" and "Q5: Contract Correctness" in the rubric so TA's do not have to scroll up and down and check two questions simultaneously.

In Fall 2021, the rough breakdown of weights for large sections are as follows:

70% Correctness

20% Handmarking

10% Testcases/Black Highlighting

*Note that these are subject to change depending on the assignment and the instructor in charge of that assignment. Towards the end of the semester, we began to focus less weight on style and more on correctness.

Once categories have been decided, set the weights for each section of correctness according to the weights given in the assignment. For example, if a question is worth 15% of correctness marks on an assignment, it should be weighted 3.75% on the rubric. If there is a bonus question on the assignment, click the “Bonus criteria” checkbox. The weight of that subsection will not be counted in the total weight.

Additionally, set handmarking and tests/cases weights according to what was decided in the marking meeting. If nothing was decided, use your best judgement to distribute marks such that the most important questions and categories are emphasized.

Fill in the respective descriptions for each rubric level according to the marking scheme. If you want a criteria to have only 3 levels (level 0, level 2, level 4), simply delete the headers and description of level 1 and level 3. For correctness and tests/cases distributions between levels, use the breakpoints.csv file for reference. Make sure to confirm with the Autotesting team, as well, that the distributions are correct.

To upload a rubric onto MarkUs, follow these steps:

<!--[if supportLists]-->1. <!--[endif]-->Create and edit the rubric using Excel. Follow the format and structure in the files attached at the bottom of the page. Do not include commas in the Excel file.

<!--[if supportLists]-->2. <!--[endif]-->Either:

<!--[if supportLists]-->a. <!--[endif]-->Export the file as a .csv

<!--[if supportLists]-->b. <!--[endif]-->Save a copy of the rubric from Teams onto your personal computer, then, using your personal Excel app, Save As a .csv file.

<!--[if supportLists]-->3. <!--[endif]-->Using the terminal, open the .csv version of the rubric and go to the end of the file. There might be extra lines filled with commas. Get rid of all extra lines. The final .csv file should have a single empty newline after the final line. Save the file.

<!--[if supportLists]-->4. <!--[endif]-->Go to the Marking Scheme tab in MarkUs, click Upload CSV, and select the file.

If this does not work, you will have to manually upload. Alternatively, instead of steps 1-3, you can create the rubric on an app that edits .csv files such as Numbers on Mac.

Fall 2020 Documentation [OLD]

Log into Markus in the admin account (the account that you log in with your email). Go to the current assignment and click on the Marking Scheme tab. Before adding any criteria, make sure that under the properties tab in the marking scheme section (Properties -> Marking Scheme), the ?rubric? option is selected.

*Note that for assignment 0, midterms and finals this may be different.

Click “Add Criterion” to add a category on the rubric. Give it a meaningful name and a weight. Note that each rubric category is marked out of 4 levels as default, so the total marks that the category will be out of should be 4 times the weight. If you want the assignment to be marked out of 100 in total, make sure the total weight is 25.0.

Note that you might find it helpful to split a criterion into two parts if you are spotchecking two different questions. For example, if you are spot-checking Q1 and Q5 for contract correctness, you can make two separate criteria titled "Q1: Contract Correctness" and "Q5: Contract Correctness" in the rubric so TA's do not have to scroll up and down and check two questions simultaneously.

In Fall 2020, the rough breakdown of weights for large sections are as follows:

70% correctness

10% purposes/correctness

10% testcases/black highlighting

10% remaining handmarking components

*Note that these are subject to change depending on the assignment and the instructor in charge of that assignment.

After splitting the weights according to this, split the weights for each subsection of correctness according to the weights given in the assignment. For example, if correctness has 7 subcategories worth 10% each, then each subcategory will be weighted 2.5 each. If there is a bonus question on the assignment, click the “Bonus criteria” checkbox. The weight of that subsection will not be counted in the total weight.

Fill in the respective descriptions for each rubric level according to the marking scheme. If you want a criteria to have only 3 levels (level 0, level 2, level 4), simply delete the headers and description of level 1 and level 3.

How To Fill In Rubric Descriptions

Correctness Criteria Descriptions

The number of correctness tests can be accessed in one of two ways. Through the test suite itself (using buildrubric, recommended) or through the most recent test run.

Buildrubric (recommended)

To use the buildrubric script, simply run the following command,

/u/cs135/markus/bin/buildrubric/buildrubric ASSN

Where ASSN is the current assignment. For example

/u/cs135/markus/bin/buildrubric/buildrubric a06

A disadvantage to using the buildrubric script is that it does not count the number of steppers, and will always treat steppers as having only 1 test.

To print out one line of the rubric with X amount of tests you can run the following command (notice the .py extension),

/u/cs135/markus/bin/buildrubric/buildrubric.py LABEL NUMTESTS

Where LABEL is the question number/label, and NUMTESTS is the number of tests in that question. For example,

/u/cs135/markus/bin/buildrubric/buildrubric.py 2a 16

Most recent test run

From the course account, the directory to access number of correctness tests for Correctness criteria descriptions is as follows:

marking -> assignment # (e.g. a01) -> ls -lt (to sort by most recent) -> test.0.studentid (take the most recent file) -> studentid -> cat OUTPUT.txt

At the very top of output.txt there should be a “total mark” section with the number of correctness tests that each question has.

In this case, question 1 has 8 correctness tests. In the rubric, each rubric level should divide the tests into categories as evenly as possible. The descriptions should be as follows:

Very Poor: No tests passed.

Weak: 1-3 out of 8 tests passed.

Passable: 4-5 out of 8 tests passed.

Good: 6-7 out of 8 tests passed.

Excellent: 8 out of 8 tests passed.

Tests/Cases Criteria Descriptions

Directories to access the number of test cases for the Tests/Cases criteria descriptions are as follows:

check-testcases -> assignment # (e.g. a01) -> test-cases -> cat filename-tc.rkt (e.g. strings-tc.rkt)

Under the -----TEST CASES----- heading, there will be a list of “make-tc”s. The number of tests will be the number of “make-tc”s that are present in that .rkt file.

To find the breakpoints for the number of test cases (breaking into different rubric levels), go to check-testcases -> refactored -> common -> cat breakpoints.csv

The first column will be the total number of tests. For example, if there are 8 “make-tc”s in A01 Q1a), then the breakpoints are 0, 2, 4, 6, 8 for the 5 different rubric levels. The descriptions for the Tests/Cases for Q1a) should then be:

Very Poor: [7-8] test cases missing.

Weak: [5-6] test cases missing.

Passable: [3-4] test cases missing.

Good: [1-2] test cases missing.

Excellent: All required cases were tested.

Rubric Template (.csv file)

There is a blank rubric template (a .csv file) attached to this page. On Markus -> Assignment -> Marking Scheme Tab, next to the “add criterion” there is an option to “Upload CSV”. Upload this .csv file to see a blank rubric template along with descriptions. Note that this is just a template - it will match the annotations template and the marking scheme on the Marking Scheme page. The marking scheme, rubric, and annotations are subject to change for each assignment.

We have also included an example excel file and an example .csv file.

Breakpoints.csv (For Reference)

Note that you can use the breakpoints formatted for the rubric attached in rubric_boundaries.xlsx below – these can be useful when making rubrics in order to avoid repeatedly typing the same thing. These breakpoints are given by:

Total,Lv 4,Lv 3,Lv 2,Lv 1,Lv 0

0,0,X,X,X,0

1,0,X,X,X,1

2,0,X,1,X,2

3,0,1,X,2,3

4,0,1,2,3,4

5,0,1,3,4,5

6,0,2,3,4,6

7,0,2,3,5,7

8,0,2,4,6,8

9,0,2,5,7,9

10,0,2,5,8,10

11,0,3,5,8,11

12,0,3,6,9,12

13,0,3,7,10,13

14,0,4,7,10,14

15,0,4,7,11,15

16,0,4,8,12,16

17,0,4,9,13,17

18,0,4,9,14,18

19,0,5,9,14,19

20,0,5,10,15,20

21,0,5,11,16,21

22,0,6,11,16,22

23,0,6,11,17,23

24,0,6,12,18,24

25,0,6,13,19,25

26,0,6,13,20,26

27,0,7,13,20,27

28,0,7,14,21,28

29,0,7,15,22,29