MarkUs Scripts

Please list scripts that have been developed for working with MarkUs, including where to find them and how to use them. If you cannot find the script on the course account, download them from the bottom of this page.

Note: The attached scripts have .txt extension for security reasons. After you download them onto the course account, remove the .txt extension.

Cut and paste the following template for each script

NAME

Location: /u/...

Purpose: Describe purpose

Example: Example Command

Notes: What can go wrong, things to watch out for, etc.

To do: Improvements and suggestions

/u/isg/bin/markus.py

This script lets you perform and automate various MarkUs tasks in the Terminal. It should be run from the course account. The script takes several sub-commands. Run /u/isg/bin/markus.py without any arguments to see a help message.

The script does some logging in the file /u/csNNN/.markus_api_script.log.

Options for all sub-commands

There are some options that can be used with all the sub-commands:

• --help, -h: Prints out a help message.
• --api-key API_KEY: Path to file containing the MarkUs API key. The default path is ~/.markus_key. The API key can obtained from the Dashboard tab in MarkUs.
• --markus-instance MARKUS_INSTANCE: The MarkUs instance you want to use. For example, if you're ISAing CS246 Winter 2018 but you want to start setting up CS246 Spring 2018, you can use --markus-instance markus_cs246_s.
• --preview: Use this option to preview what MarkUs will do. If this option is used, the script will not actually modify anything in MarkUs.
• --nocache: By default, the script will reuse data that it downloads from MarkUs. This makes the script faster, but can lead to outdated results. If you specify the --nocache option, then the script will always download the latest information from MarkUs. This can make the script much slower.

upload_marking sub-command

You can use this sub-command to upload files to MarkUs for marking, or to change files after a submission has been collected. This sub-command only works if the submission has already been collected (the row is green in the assignment's Submissions tab). If you want to change a file that's already in MarkUs, use the --overwrite or -o option, but beware that annotations made on the file will be deleted.

Example 1: /u/isg/bin/markus.py upload_marking a01 yc2lee /u/cs123/yc2lee_code.rkt a01_code.rkt

This command will upload file /u/cs123/yc2lee_code.rkt to yc2lee's marking page for assignment a01 (the assignment short ID). In the marking page, in the drop-down menu of files, the file will be named a01_code.rkt.

Example 2: /u/isg/bin/markus.py upload_marking --match 'GRADED_ASSIGNMENT.py' a01 /u/cs123/handin/a01_autotest/

In this example, the /u/cs123/handin/a01_autotest/ folder contains a folder for each student. Inside each student's folder is a file named GRADED_ASSIGNMENT.py that you want to upload. This command will upload the GRADED_ASSIGNMENT.py file for each student. The --match argument is a regular expression, and files whose path matches the regular expression will be uploaded.

delete_marking sub-command

This sub-command deletes the marking files for a student. Once the marking files are deleted, when you go and mark the student's work, nothing will appear in the drop-down menu of files to mark. The student's submissions, ie. their repository which keeps track of each submission, is unaffected.

Example 1: /u/isg/bin/markus.py delete_marking a01 yc2lee a01_code.rkt

This will delete the file a01_code.rkt for yc2lee for assignment a01. MarkUs will search for the file in the list of files in the drop-down menu you see when marking.

Example 2: /u/isg/bin/markus.py delete_marking a01 yc2lee

This will delete all marking files for yc2lee for assignment a01.

download_marking sub-command

This sub-command downloads the files that are being marked, ie what you see when you are marking. It does the same thing as the "Download all files" button on the marking web page.

Example 1: /u/isg/bin/markus.py download_marking 'Assignment 01' a01_marking_backup

This downloads all the students' files used for marking Assignment 01, and saves them in the a01_marking_backup folder.

Example 2: /u/isg/bin/markus.py download_marking --annotations 'Assignment 01' yc2lee yc2lee_marking_backup

This downloads yc2lee's Assignment 01 files that are used for marking, along with any annotations made. The files are saved in the yc2lee_marking_backup folder.

upload_svn sub-command

Use this sub-command to upload files to a student's SVN repository. Files uploaded to a student's SVN repository will be immediately visible to the student (unless the assignment is hidden). Uploading to a student's SVN repository is like submitting or uploading files for the student, except the public test is not run.

This sub-command uses the MarkUsGroupsSVNRepos#u_markus_svn_add_file and MarkUsGroupsSVNRepos#u_markus_svn_add_dir tools.

Example 1: /u/isg/bin/markus.py upload_svn a01 yc2lee /u/cs123/yc2lee-files/

This command uploads the files in /u/cs123/yc2lee-files/ to yc2lee's SVN repository for a01.

Example 2: /u/isg/bin/markus.py upload_svn a01 /u/cs123/handin/marmoset_a01/

In this example, the /u/cs123/handin/marmoset_a01/ folder was downloaded from Marmoset. The folder contains a folder for each student, with each student's files inside his or her folder. This command will upload all the files to each student's repository.

delete_svn sub-command

Use this sub-command to delete files from the student's SVN repository. This sub-command uses the MarkUsGroupsSVNRepos#u_markus_svn_delete_file and MarkUsGroupsSVNRepos#u_markus_svn_delete_all tools.

Example 1: /u/isg/bin/markus.py delete_svn a01 yc2lee a01q1.rkt

Deletes yc2lee's a01q1.rkt file from his SVN repository in assignment a01.

Example 2: /u/isg/bin/markus.py delete_svn a01 yc2lee

Deletes all of yc2lee's files in his SVN repository in assignment a01.

download sub-command

Use this sub-command to download student's submissions from MarkUs. You can use the following optional arguments to specify at what time you want to download the submissions from:

• --due-date: (default) Download the last submission before the due date in Markus. Does NOT use the collection time from the Submissions tab; for that, see download_marking sub-command
• --time 'YYYY-MM-DD hh:mm:ss': Download submissions from the given time. For example, to download the latest submissions made before April 2, 2018 8:30AM, use --time '2018-04-02 08:30:00'.
• --latest: Download the latest submissions, even late ones
• --revision: Download the submission specified by revision number

Example 1: /u/isg/bin/markus.py download a01 /u/cs123/handin/a01_autotest

This example downloads all the on-time a01 submissions to the /u/cs123/handin/a01_autotest folder.

Example 2: /u/isg/bin/markus.py download --time '2018-04-03 15:30:00' a01 yc2lee /u/cs123/yc2lee-files

This example downloads yc2lee's last submission before April 3, 2018 3:30PM. The files will be saved in the folder /u/cs123/yc2lee-files.

create_group sub-command

Use this sub-command to create groups from the Terminal. You can create also create groups from the Groups tab in MarkUs.

Example 1: /u/isg/bin/markus.py create_group a01

This example will create a group for each student for a01. The students to create groups for are obtained from the Users => Students tab, so you should add all the students to MarkUs before running this command.

Example 2: /u/isg/bin/markus.py create_group --repo-name blue_team_repo a01 blue_team yc2lee j9smith

Creates a group called blue_team, with repository blue_team_repo. The group will have students yc2lee and j9smith.

delete_group sub-command

Use this sub-command to delete groups.

Example 1: /u/isg/bin/markus.py delete_group a01

Deletes all groups for assignment a01.

Example 2: /u/isg/bin/markus.py delete_group a01 yc2lee

Deletes the group yc2lee from a01.

set_mark sub-command

Use this sub-command to fill in the mark for one criteria for one student. If the assignment is using rubric marking scheme, the mark is the level.

Example 1 using flexible marking scheme: /u/isg/bin/markus.py set_mark a04 yc2lee "Q1 Correctness" 16.5

This example gives yc2lee a mark of 16.5 for the criteria "Q1 Correctness" in assignment a04.

Example 2 using rubric marking scheme: /u/isg/bin/markus.py set_mark a05 yc2lee "Q1 Correctness" 3

This example gives yc2lee level 3 for the criteria "Q1 Correctness" in assignment a05.

set_marks_csv sub-command

This sub-command fills in marks using data from a CSV file that you have to create. The CSV file should have 3 columns:

• Column 1 is userid or the group name from the Submissions tab in MarkUs
• Column 2 is the name of the criteria you want to set a mark for.
• Column 3 is the mark/level to give.

Example: /u/isg/bin/markus.py set_marks_csv a03 a03_csv_marks.csv

This will fill in a03 marks using a03_csv_marks.csv. As an example, a03_csv_marks.csv might contain:

yc2lee,Q1 Correctness,1
yc2lee,Q2 Correctness,4
yc2lee,Q2 Test/cases,2
g9yang,Q1 Correctness,4
g9yang,Q2 Correctness,3

In this example, yc2lee would get Level 2 for the criteria "Q2 Test/cases".

set_marks_rst sub-command

This sub-command fills in marks using RST test results.

The second argument rst_results_folder is the path to the folder created by RST (the folder is named something like test.1.AUTOTESTRESULTS). The folder contains a folder for each student containing their test results. By default, the sub-command will look for test results in a file named OUTPUT.txt. You can change this by using the --rst-results-filename argument. The OUTPUT.txt files must contain the total for each question, for example something like this:

 ** Question 1: 12/13
 ** Question 2: 1717
 ** Question 3: 15/17

If these totals are missing from your OUTPUT.txt, try adding (print-by-question true) to config.ss. Then run RST again, and check that OUTPUT.txt now contains each question's total mark.

The third argument rst_to_markus_csv is a CSV file that tells the script which criterion in MarkUs corresponds to each RST question.

Flexible marking scheme: Each row in the CSV file has two columns. The first column is a RST question name. In the OUTPUT.txt created by RST, each question's mark is printed in a line like " ** Question N: X/Y". The first column in the CSV file is the N.

The second column is the name of the corresponding criterion in MarkUs. The script will give students whatever mark they got in the RST test results.

Example using flexible marking scheme: /u/isg/bin/markus.py set_marks_rst a01 ~/marking/a01/test.1.AUTOTESTRESULTS ~/marking/a01/flexible_converter.csv

In this example, ~/marking/a01/test.1.AUTOTESTRESULTS is the folder created by RST. The example ~/marking/a01/flexible_converter.csv file:

1,Q1 Correctness
2,Q2 Correctness
3,Q3 Correctness

For example, if yc2lee's OUTPUT.txt contains " ** Question 1: 14/17", then yc2lee will get 14 for the criterion "Q1 Correctness".

Rubric Marking Scheme: The CSV file says which level to give depending on the RST score. Each row in the CSV file has 6 columns:

1. First column is a RST question name (the N in " ** Question N: X/Y" from the OUTPUT.txt file).
2. Second column is the name of the corresponding criterion in MarkUs.
3. Third column is the minimum mark needed to get Level 1.
4. Fourth column is the minimum mark needed to get Level 2.
5. Fifth column is the minimum mark needed to get Level 3.
6. Sixth column is the minimum mark needed to get Level 4.

If a level is not used, put in "X" or leave the column empty instead of entering a number.

Example using rubric marking scheme: /u/isg/bin/markus.py set_marks_rst a02 ~/marking/a02/test.1.AUTOTESTRESULTS ~/marking/a02/rubric_converter.csv

In this example, ~/marking/a02/test.1.AUTOTESTRESULTS is the folder created by RST. The example ~/marking/a02/rubric_converter.csv file:

1,Q1 Correctness,2,4,6,7
2,Q2 Correctness,X,1,X,2
3,Q3 Correctness,1,2,,4

The first row says you need 2 marks to get Level 1, 4 marks for Level 2, 6 marks for Level 3, and 7 marks for Level 4.

The second row says you need 1 mark for Level 2, and 2 marks for Level 4. Levels 1 and 3 are not used.

The third row says you need 1 mark for Level 1, 2 marks for Level 2, and 4 marks for Level 4. Level 3 is not used.

set_marking_status sub-command

Use this sub-command to set/update the marking status. The submission needs to already be collected.

Example 1: /u/isg/bin/markus.py set_marking_status a01 yc2lee complete

Sets yc2lee's a01's marking status to complete. This only works if all the marks are filled in for yc2lee.

Example 2: /u/isg/bin/markus.py set_marking_status a01 complete

Sets the marking status for all a01 submissions to complete. An error will be printed for students that are missing marks.

Example 3: /u/isg/bin/markus.py set_marking_status a01 unmarked

Sets the marking status for all a01 submissions to unmarked.

submission_info sub-command

Shows information about a student's submissions.

Example 1: /u/isg/bin//markus.py submission_info a01 yc2lee

Prints information about yc2lee's a01 submissions.

Example 2: /u/isg/bin//markus.py submission_info a01 yc2lee --latest

Prints information about yc2lee's last a01 submission.

Example 3: /u/isg/bin/markus.py submission_info --json a01 yc2lee

Prints the information in json format for easier automated processing.

svn_add_file_stdin, svn_add_file, and svn_add_dir

Location: /u/markus/svn_add_file_stdin, /u/markus/svn_add_file, /u/markus/svn_add_dir

Purpose: These scripts are used to add file(s) to students' MarkUs account. Run them from the course account in a terminal. Documentation and examples are at MarkUsGroupsSVNRepos#Terminal_Adding_files_from_cours

parseStudents.py

Location: /u/csXXX/csXXX-sources/parseStudents/

Purpose: This script is used to add students into MarkUs and Piazza without the need to run multiple scripts. It produces two files, one fore MarkUs (markusClassList.csv), and one for Piazza (piazzaClassList.csv). For the MarkUs file, it will print the Quest ID and name of each student. For the Piazza file, it will print the email address of each student. These two files can be uploaded onto MarkUs and Piazza.

During the first month of the term, you should run this script and re-upload the printed output onto MarkUs and Piazza daily.

Example: This script should be run from the course account.

cd /u/csXXX/bin/
python parseStudents.py

This command should produce two files under csXXX-sources/ParseStudents titled markusClassList.csv and piazzaClassList.csv.

Notes: Make sure you run this script frequently at the beginning of the term, so that students who register late will be added.

(Edited from parseStudentsToLoadIntoMarkus.py)

CheckoutAndCommit.py

Location: /u/csXXX/markus/bin/CheckoutAndCommit.py

Purpose: Used to checkout the student's submissions, and to add the GRADED_ASSIGNMENT.ss file. See MarkUsGroupsSVNRepos#Auto_testing_Procedure for more information, including when to run this script.

There are five variables in the script that you have to set. The script itself describes the variables and how to set them.

This script also expects two arguments.

• The first argument can be either checkout or commit.
  • If you use checkout, this script will checkout the students' code to checkout_folder (one of the variables you have to set).
  • If you use commit, this script will add and commit GRADED_ASSIGNMENT.*
• The second argument can be either viewcmds or runcmds.
  • If you use viewcmds, this script will display some terminal commands, but it won't execute any of them.
  • If you use runcmds instead, then this script will execute the commands.

Examples: Here is an example from Fall 2011, CS 115, Assignment 04. The assignment was due at 9:00pm on October 26, 2011.

Example values for the variables inside the script:

• csv_file_location = '/u/cs115/marking/a04/a04groups.csv'
• assn = 'a04'
• checkout_folder = '/u/cs115/handin/a04_autotest'
• due_date = '2011-10-26 21:10'
• markus_url = "markus.student.cs.uwaterloo.ca/markus_cs115_f/en/main"

This script can be run with one of the following commands:

• python CheckoutAndCommit.py checkout viewcmds
• python CheckoutAndCommit.py checkout runcmds
• python CheckoutAndCommit.py commit viewcmds
• python CheckoutAndCommit.py commit runcmds

MakeGradedAssignment.py

Location: /u/csXXX/markus/bin/MakeGradedAssignment.py

Purpose: Joins the auto-testing results with the student's code. You can do this joining for every student, or just one student. See MarkUsGroupsSVNRepos#Auto_testing_Procedure for more information, including when to run this script.

There are five variables in the script that you have to set. The script itself describes the variables and how to set them.

You can run python MakeGradedAssignment.py --help to see a useful help message:

linux028:~/markus/bin> python MakeGradedAssignment.py --help
You can use this script to create GRADED_ASSIGNMENT.py for every student, or just one student.

1) To create for every student:
    Set the five variables at the top of the script.
    Then run this script with no arguments: python MakeGradedAssignment.py
    The script will create GRADED_ASSIGNMENT.py file for all the students

2) To create for one student:
    Run this script like this:
    python MakeGradedAssignment.py assign_nbr quest_id autotest_results file1 file2 ... filen
        assign_nbr: The assignment number. Ex: 06
        quest_id: The student's Quest ID. Ex: jsmith
        autotest_results: Path to OUTPUT.txt file produced by RST
        file1 file2 ... filen: Path to the  student's files. Ex: a06q1.rkt a06q2.rkt a06q3.rkt
    When run like this, the script will ignore the variables set
    in the script, and instead use the arguments  provided.  The
    output will be printed to the screen. No files will be created
    or modified.

Note: If you enter .rkt files, the script will first try to find the
      exact file you entered. If it cannot find the file, it will try
      the .ss and .scm extensions.

Examples: Here is an example from Fall 2011, CS 115, Assignment 04. The test suite for RST is called test.1, and the run_id used is AUTOTESTRESULTS.

Example values for the variables inside the script:

• assn_name = '04'
• file_list = [ 'a04q1.rkt', 'a04q2.rkt', 'a04q3.rkt', 'a04q4.rkt' ]
• assn_folder = 'a04_autotest' # note that it's not a04
• autotest_folder = 'test.1.AUTOTESTRESULTS'
• output_filename = 'GRADED_ASSIGNMENT.ss'

If you want to make GRADED_ASSIGNMENT file for every student, run python MakeGradedAssignment.py with no arguments. The sample output below has been truncated.

linux028:~/markus/bin> python MakeGradedAssignment.py 
Creating GRADED_ASSIGNMENT.ss for the 43 students in
/u/cs115/handin/a04_autotest/
Test results will be obtained from
/u/cs115/marking/a04_autotest/test.1.AUTOTESTRESULTS/

Continue? (y/n): y
OK! Making files...
--- Creating GRADED_ASSIGNMENT.ss for acschott-student --- 1/43
Creating /u/cs115/handin/a04_autotest/acschott-student/GRADED_ASSIGNMENT.ss
Autotest results /u/cs115/marking/a04_autotest/test.1.AUTOTESTRESULTS/acschott-student/OUTPUT.txt
--- Creating GRADED_ASSIGNMENT.ss for h9stud --- 2/43
Creating /u/cs115/handin/a04_autotest/h9stud/GRADED_ASSIGNMENT.ss
Autotest results /u/cs115/marking/a04_autotest/test.1.AUTOTESTRESULTS/h9stud/OUTPUT.txt

You can create the GRADED_ASSIGNMENT file for one student only. This is useful if students submit late or you're remarking. If you run

python MakeGradedAssignment.py 04 h9stud ~/marking/a04/test.1.AUTOTESTRESULTS/h9stud/OUTPUT.txt a04q1.rkt a04q2.scm a04q3.ss a04q4.ss

then the script will create the GRADED_ASSIGNMENT file for h9stud, printing the results to the screen. You can save the output:

python MakeGradedAssignment.py 04 h9stud ~/marking/a04/test.1.AUTOTESTRESULTS/h9stud/OUTPUT.txt a04q1.rkt a04q2.scm a04q3.ss a04q4.ss > GRADED_ASSIGNMENT.ss

Notes: Suppose you make the GRADED_ASSIGNMENT files for every student, and then realize the GRADED_ASSIGNMENT files are messed up. It's easy to delete all the GRADED_ASSIGNMENT files:

cd /u/cs115/a04_autotest/
rm */GRADED_ASSIGNMENT.ss

Once the GRADED_ASSIGNMENT files have been deleted, you can fix anything that needs fixing and rerun MakeGradedAssignment.py.

markerallocation.py

Location: Download markerallocation.py in table below

Purpose: This script assigns submissions to graders

Usage: This script consumes two arguments. You can run it like this: python3 markerallocation.py startmap marking_counts

• start_map: The CSV file downloaded from MarkUs Graders tab. This is a list of students that we have to assign to graders. It can contain mappings between students and graders, and by default they will be kept.
• marking_counts: This is the path to a CSV file you have to create. In the CSV file, each line should be like this: graderid,X where graderid is a grader's Quest ID, and X is the number of assignments that grader has to mark.

The script prints a student-grader map to the screen. You can save the output in a file and upload it onto MarkUs.

Example: Here's an example markinglist.csv, where there are 103 submissions to mark:

a9grader,30
a2grader,30
a4tutor,10
a23prof,3
a10ta,30

Here, a9grader has to mark 30 assignments and a4tutor has to mark 10. If startmap.csv is the file you downloaded from MarkUs, run this script with

python3 markerallocation.py startmap.csv markinglist.csv > finalmap.csv

Then go back to the "Graders" page and upload finalmap.csv.

Notes: The number of submissions to mark must equal the sum of the numbers in markinglist.csv. In the example above, if there were only 100 submissions that needed to be marked, the script will produce an error:

Counting Error:
There are 100 students/groups that have to be assigned to TAs
But the sum of the numbers in markinglist.csv is 103

computeMarks

Location: /u/csXXX/marking/test.pt/computeMarks

Purpose: This script distinguishes between the "all in one file" and the "individual file" submissions to markus.

Example: For an assignment students can submit just one file a3.pdf or they can submit 2 different parts: a3p1.pdf, a3p2.pdf.

If they submit a3.pdf the following message is emailed:

**************************************
CS240 Winter 2013
Assignment 3 Public Test

This public test checks which files you have submitted.
It does not check the files' contents in any way.

Found the file a3.pdf.
* End of files *********************

If they submit only a3p1.pdf the following message is emailed:

***************************************
CS240 Winter 2013
Assignment 3 Public Test

This public test checks which files you have submitted.
It does not check the files' contents in any way.

We could not find a3.pdf.
Searching for individual files ind1.pdf ind2.pdf ind3.pdf ind4.pdf

a3p1.pdf found.
a3p2.pdf could not be found!!!
*** End of files ********************

Notes: To change which files to look for simply change the all_in_one and indv_files variables in the copmuteMarks script.

To do: Could add more descriptive messages.

#copmuteMarks

---

buildrubric

Location: /u/cs135/buildrubric

Purpose: Produces a csv style output of the questions and mark ranges in a given assignment and term, using the correctness test folder to calculate the mark ranges. It also appends on a generic marking scheme for style/purpose/contracts/tests/examples etc.

Example: ./buildrubric a06 1149

=>

Notes:

• The correctness tests must be complete (or at least the folders for them) in order for this script to work, since it counts the number of folders in each question's correctness tests folder.
• The generic rubric is called "general-rubric.csv" and is also located in the /u/cs135/buildrubric folder

In order to use this script, you may have to modify the directory path it accesses to be correct for your course. The default file is for cs135.

To do:

• Could add more options to modify the generic rubric.

-- JenniferLuu - 14 Jan 2013

• computeMarks.txt: computeMarks script