Resources and Information for CS 240 ISAs

This twiki is for technical details only. Any other duty information can be found in Waterloo LEARN (D2L): CS Instructional Support Group -> Course Admin & Duties for ISAs.

Important Info [cs240E]:

• Information For CS240E ISAs

Timeline

First Day:

• Enter the course account and familiarize yourself with the locations of files, the directory structure, all the HTML files under public_html.
• Upload assignment 0 and assignment 1 to the course webpage after checking the questions and verifying that it's the final version with the prof.

Week 1:

• After becoming familiar with the course account, go onto MarkUs and familiarize yourself with creating and modifying assignments, marking schemes, adding students, and managing graders. Provide a list of the file names that students are expected to submit, but allow for any file names to be accepted to account for typos.
• Go on Piazza and make announcements about assignments 0 and 1 being released.
• Make the weekly post on Piazza about upcoming deadlines and an overview of this week + next week. Look in archives for references on how to format it and what to include.
• Add modules 0 and 1 to the course website (see Uploading Modules to the Course Website below).
• Familiarize yourself with the course email account and reply to any relevant emails so far (see Email Guidelines below).
• Add new students to Markus, Piazza and Marmoset (every day for the first few weeks).
  • For Markus, go to Users->Students and click "Upload from course account".
  • For Piazza, run (odyssey classlist | cut -f2 -d, | sed 's/.*/&@uwaterloo.ca/' |sed '1d' > piazzaClasslist.csv) in the course account to generate a csv in the right format. Then go to Manage Class->Enroll Students in Piazza and upload it.
  • For Marmoset, follow the instructions in the Instructor View of a course, People->Register students or groups for this course using a CSV file.
• Update the course website with all relevant details for the new term. Let ISC/instructors know if something is missing.
  • Check all pages and links (e.g. correct term and year, dates)
  • Update list of course staff (+ pictures) and assignment deadlines
  • Update lecture and tutorial times (can see the Schedule of Classes for information)
  • Previous terms can be used as a reference.
• At the end of the week, prepare for tutorial 0 by familiarizing yourself with LaTeX.

Week 2:

• Post your office hours (as well as IAs' and instructors') on the Google Calendar and make sure to attend them.
  • See ~/.googlelogin in the course account for login details.
• Conduct your first tutorial and add tutorial 0 slides/pdf/tex file to the course webpage (similar to adding modules).
• Learn the procedure to collect assignments and how to use markerallocation.py in ~/markus/bin to assign graders to student submissions. (You can also do this in MarkUs by clicking "Users -> Graders")
• Check MarkUs Scripts on ISG Twiki to see how the course's scripts are used.
• Meet with the TAs to discuss marking assignment 0.
• Ask for assignment 2 from the prof and start doing its questions.
  • Check for typos and any complicated language that you think could be made clearer.
  • Make sure the assignment is a good length. Students should be spending ~5 hours a week on assignments, ISAs less.
  • Make sure the assignment is a good difficulty. Hints can sometimes be good for a difficult question.
  • Check for alternative solutions to questions.

Week 3-4:

• Upload the marking scheme and annotations for assignment 1 to MarkUs after receiving it from the prof -After assignment 1's due date has passed, release assignment 2 as quickly as possible.
• An hour after assignment 1's due date has passed, upload solutions to the course webpage by putting the solution file in ~/protectPDF/cs240a1solns and modifying ~/public_html/TERM/protect/requestAssgnSolnCommon.php -Learn the autotesting procedure for upcoming assignments that have programming questions.
• MarkUs does not handle autotesting. You must create your own bash script to test students' submissions.
• Meet with the instructor to discuss the test cases that they want.

Continually throughout the term:

• Prepare for tutorials.
• Attend lectures and tutorials
• Attend your office hours to help students
• Do the assignments the prof gives you before you release them (proofread)
• Collect assignments about 10 minutes after their due date
• Run markerallocation.py to assign TAs to submissions (You can manually do this now since MarkUs introduces a new feature)
• Host marking meetings with TAs to discuss marking
• Manage programming questions when required
• Upload marking schemes and annotations to MarkUs
• Upload assignment solutions an hour after the due date

Week 8 is usually the midterm. You'll have to proctor it and then do the midterm marking. Similarly for the final exam.

If you want to access the course folder when not using the office computer, here is a guide you can follow: https://cs.uwaterloo.ca/twiki/view/ISG/SambaWindows.

Uploading Modules to the Course Website

1. Pull the latest changes from the git repository in ~/modules (i.e. git pull).
2. Run ./buildHandout.pl xx where xx is the module number to build. (e.g. ./buildHandout.pl 00 to build module 0). This will generate two files, modulexx.pdf and handoutxx.pdf.
3. Copy both files to ~/public_html/w22/modules.
4. Make sure the files have read permissions for others in order to be viewable on the website. If not, run chmod o+r to add the permission.
5. Modify lectures.phtml to link to the modules.

Assignments

Setup

Markus

• Short identifier: Should be exactly A# (one digit) so that it matches with the public test directories on the course account
• Additional options:
  • Allow web submissions.
  • Display submission and revision history to students.
  • Hide the assignment from students at first, and uncheck when the assignment is released.
  • All other checkboxes should be unchecked.
  • [Old] If testing using RST, let students run and view basic/public test results (they also receive emails, but this is a nice alternative way of viewing the test results).
• Required Files: Generally specified in the assignment, but for A0 it should be "a0.pdf" and "a0.tex", and for A1-5 it should be of the form "a#.pdf" (for single file submissions -- if allowed) and "a#q#.pdf" for each question (the second number being question number).
• Submission Rules: Late submissions receive no marks.
• A0 of CS 240 always uses a rubric marking scheme (i.e. NOT flexible). Most assignments after A0 use a flexible marking scheme. If the assignment has not changed from the previous term, upload the rubric used in the previous term. If it has changed, either modify the csv file from the previous term and then upload (this can be easier if there are many changes) OR upload the previous term's rubric and modify it in MarkUs. Or alternatively, start from scratch; there isn't usually a lot to do.
• Likewise to above for annotations. Annotations can be created as the assignment is marked, but for this to be possible there at least need to be annotation groups created (suggested: one called "General" and one for each question). Uploading annotations from a previous time a question was used can be a useful starting point, though note that you should probably go through them and remove duplicates, fix spelling/punctuation as necessary etc.

Marking Scheme

• Get the sample marking scheme from instructors. This will depend on the instructor, but is often a very bare-bones marking scheme which you will need to flesh out.
  • Include the questions and the mark distribution of each question.
  • Provide details on where and when part marks can be given.
  • Ensure that it covers alternative solutions.
  • Add any clarifications to streamline marking.
• The modified marking scheme should be typed up (ideally in LaTeX, see archives for examples) and the updated version should then be sent to the instructors to ensure they are okay with your modifications.

BEFORE marking meetings:

Before the very first marking meeting, TAs need to be added as Graders in MarkUs (see ISG Twiki for how to). The ISC will set up the first marking meeting; the rest will be set up by ISAs.

For future marking meetings:

1. Set up Markus (see above). If there are bonus marks, the criteria in Markus MUST have "Bonus" in the criteria name. This is to make the marking clearer to students and for the processing later that results in the web page marks posting (so bonus marks are correctly taken into account in what the assignment is out of).
2. Mark 2 or 3 to see if the marking scheme and/or annotations need refining (and if so, check with the instructor) and to help with the next point.
3. Create a marking notes document (see above).
4. Set up a dedicated channel on Teams and set up a meeting. It is helpful to send the invitation at least a couple of days prior, or schedule a recurring meeting so TAs have the meetings scheduled in their calendars. In Teams, post the list of topics covered as well as the assignment + solutions (note solutions are confidential) so TAs can have a chance to review them before the meeting.
5. Assign students to graders. See the ISG Twiki, https://cs.uwaterloo.ca/twiki/view/ISG/MarkUs#Assign%20Submissions%20to%20Graders, for instructions. If the markerallocation.py script is not available, let the ISC know. The ISA should mark about 10-15 for A0 and 5-10 for other assignments. Remember to make sure if there are any double TA units since they get twice the marking as single units.
   • Assign all students to all TAs (except for the students that the ISA will mark), and create a spreadsheet assigning TAs to specific questions. If there are more TA units than the number of questions, split students into equal groups for certain TA units to mark a question for only one of these groups of students. The easier way to group students off is by giving ranges of student quest IDs (as submissions are ordered alphabetically by student quest ID) so that each range has as close to the same number of students as possible. Make sure to always split students for the more difficult questions before the easier ones in case TA units do not divide evenly into the number of questions.
   • Rotate TAs between easy and difficult questions throughout the term, and also note the amount of time they report they spend marking previous assignments to even out the hours each TA spends on future question assignments.
   • [old: TAs should not have more than 35 students to mark each assignment for a single TA unit and the ISA marks any excess if TAs are at 35 (the ISA should always be marking 5-10 and if that leaves <35 each for TAs, that is fine).]

Post sample solutions (protected, students authentication required -- see protectPDF/README.txt on the course account) an hour after an assignment deadline (not needed for A0).

The following information is from when marking meetings were done in-person (old):

At least 7 hours before (preferably the previous day) a marking meeting: send an email to the graders (cc: cs240@uwaterloo.ca and the instructor) with Subject: [cs240] Assig x marking meeting reminder (fill in "x" appropriately) and remind them of the meeting day, time and location, and to bring their laptops (if possible) and to go through the assignment beforehand.

The ISA will prepare sample solutions to hand out to each grader (also copies for the ISA(s) and ISC) at the meeting and to email to graders AFTER the marking meeting (we want to make sure they come to the meeting to get the important verbal notes).

Also include in the marking package:

• a printout of the Assignment questions
• Make sure no marking notes are inside the sample solution
• for A0: a printout of the LaTeX file (for reference to see if students have made changes)
• Sample solution note: A1 and on: if the instructor supplies sample solutions, check that they meet all assignment guidelines. If the ISA is providing, have the instructor check them. These sample solutions will also be the ones posted in the protected area (student authentication required). However, note that these solutions are not always full solutions (i.e. the detail that is expected of students); they can just be solution ideas.

DURING the marking meeting:

At the very first marking meeting, the ISC will say a few words of welcome and other notes about expectations. They won't show up for future marking meetings; the ISAs will lead those on their own. Take attendance at every meeting and after the last meeting, give a copy of the attendance records to the ISC (keep the original to help with TA evaluations).

The ISAs will go through the general points of the marking scheme and expectations for the answers, and especially important notes that may be in the marking notes but that we want to make sure the graders don't miss. It is helpful to go through a student's solution that has already been marked as an example (better if it has some mistakes).

TAs should be instructed to complete their first 5 submissions ASAP (preferably before the weekend) so that you can give feedback before they start marking all of them.

AFTER the first marking meeting:

Feedback should be given via email from the course account. You can view a TA's submissions by "switching" your user to their account in MarkUs. Some things to look for when giving feedback:

• following the marking scheme
• always leaving annotations when marks are deducted
• if annotations are unclear or incorrect
• anything that was marked incorrectly (e.g. alternate solution, misinterpretation of student's answer)

Any students that submit late and send an email to the course account can have their assignment marked for feedback only (not for marks), except in special cases. Don't forget to apply the deduction at the end.

The ISA or the instructor (or if multiple instructors, the instructor responsible for this assignment) should be answering any marking questions from TAs. It will be the ISA's job to act first and not wait for the instructor to respond but if you are unsure how to answer, send the instructor a separate email to inquire.

The day before the marking deadline, check if any TA has completed less than 10 submissions. If so, send them an email reminder to make sure they know about the deadline and have an opportunity to ask for help if they need it.

Once they are done, TAs should reply to the appropriate threads in Teams with their marking hours and any common errors they found. If they do not do this, send a reminder as both items are important: the hours for tracking and the errors for creating the post-mortem. Examples of previous post-mortems are in the archives. Have the instructor check it over (make any suggested modifications), then post when you release the marks. If the post-mortem is not ready shortly after marking is done, it is okay to release the marks first but the post-mortem should be up within a day of that. The assignment average should be taken from MarkUs and displayed on the course website. Let the ISC know when marking is done so the ISC can get the web page marks posting updated.

Communication via Email (old):

Email the marking notes and sample solutions to the graders (cc: cs240@uwaterloo.ca and the instructor) with Subject: "[cs240] A# Marking Documents" (fill in "#" appropriately) . Also note that graders should reply all to this email whenever they run into marking questions so everyone will be able to be fair and consistent in marking and that graders should email cs240@uwaterloo.ca when done marking and with the common problems encountered. See course email for past emails that could be used as a template.

Once marks are released:

Keep track of which TAs get the marking done on time and which don't (including how many days late, as well as reasons, if any given, for the delay) as well as any marking quality issues. This will also be used as part of TA evaluations.

Remark requests are only handled 2 weeks after marks are released. The student requesting a remark must send an email with their quest ID, indicate which questions they are requesting a remark for and why. To remark a submission:

1. Find the user's submission on Markus.
2. Review the marking scheme and solution to see if a remark is deserved. If the case is unclear, bring it up with the instructors.
3. If a remark is deserved:
   1. Unrelease the solution by unchecking the checkbox.
   2. Change grade values, leaving annotations where appropriate.
   3. Mark the submission as complete and release it again. (Don't forget this step!)
4. Regardless of whether a remark occurred or not, reply to the student with why you took the actions you did.

MOSS:

• We use Moss to detect plagiarism for all programming assignments
• The instructions for running Moss can be found at MossBasics
• Note that the ISA needs to run Moss right after the deadline for each programming assignment.

Academic Integrity Declaration Check:

You want to check if students have submitted the academic integrity agreement, signed it properly, and not altered the rest of the text of it after the assignment deadline has passed.
The scripts for checking these things are in /u/cs240/bin/acadinteg (or /u/cs240e/bin/acadinteg). Following are the steps to do these checks:

• For written assignments, wait until the assignment guidelines have passed. For programming assignments, after the deadline has passed, you then need to download all submissions, academic integrity agreements, and grades first from Marmoset (how to above).
• If it does not already exist, create a directory acad-integ-checks/ inside ~/handin
• Create a directory for the assignment itself: acad-integ-checks/AX (for example acad-integ-checks/A1/ or acad-integ-checks/P1/)
• Copy the academic integrity agreement which was posted on the website for students for that assignment into acad-integ-checks/AX. Let's call it AX-AID.txt.
• cd inside the acad-integ-checks/AX directory if you have not already
• Call the checking script as follows ~/bin/acadinteg/check-assn-acadinteg.sh ~/handin/AX ~/handin/AX AX-AID.txt ax_aid_results.txt
• The first argument is the directory which has the student submissions for their assignment
• The second argument is the directory which has academic integrity agreements submitted by the students. This is the same as the first argument for written assignments. However, for programming questions, you will download marmoset assignment submissions to ~/handin/PX and the academic integrity submissions to ~/handin/PX-AID. The reason we need the first argument is because we want to check the academic integrity form submissions for students who actually submitted the assignment itself, so we need to know who submitted the assignment. So for programming questions our command will look like: ~/bin/acadinteg/check-assn-acadinteg.sh ~/handin/PX ~/handin/PX-AID PX-AID.txt px_aid_results.txt
• The third argument is the sample academic integrity file we provided for that assignment to the students, so that the script has something to compare against
• The last argument is the destination file where it will print all the results to
• After you have your results, you need to manually check through the flagged usernames. This is because sometimes auto slashes are inserted in files for some reason, or some latex inserted metadata if the student opened the academic integrity agreement in latex. Students sometimes also make the agreement into a pdf which causes them to get flagged. Students may also have submitted after the deadline after Marmoset submissions have been downloaded, but we are okay with late academic agreements as long as it is there when we do these checks. You may also want to check the MarkUs repository for late ones during your manual checks. So for all these minor issues, we don't need to have them resubmit the agreement. We do want students to resubmit the agreements when:
  • There is a significant altering of the text of the agreement
  • Agreement is unsigned
  • There is a typo in the spelling of the userid
  • Student used their email id or student number to sign the agreement
  • Student submitted the agreement for a different assignment
  • Any other major issues with the agreement
• Once we have manually checked and narrowed down the userids to students with major issues as mentioned in the previous point, we want to send each student an email with the following template (You can BCC them if that makes it easier, but be very careful to protect student confidential information)

  ---

Dear Student,

You are getting this email because you did one of the following:

* Did not submit your AX academic integrity form or submitted a corrupted file
* Submitted the form with the wrong or missing signature - you must use your WATIAM userid. It should not be your student number and it should not be your email. It must not have any typos.
* Submitted the wrong form or with alterations (outside of the userid field) made to it.


Please submit the correct agreement to MarkUs in the next 24 hours, or your AX mark will be set to 0.
Please contact Karen Anderson, Instructional Support Coordinator (kaanders@uwaterloo.ca) if you have any issues with this.

Regards,
Your Name
CS240 ISA

---

Be sure to change Markus to Marmoset, and the assignment names accordingly based on the assignment.

• After 24 hours have passed from the above email, recheck the students you sent the email to if they submitted the correct agreement after.
• For the students that still have not submitted the agreement by this point, send their names to Karen, along with the issue with their agreement. You do not need to do anything else and do not need to change any marks.

Tips on conducting tutorials:

• On the first day, introduce yourself to make students feel more comfortable (e.g. also another student, just on coop).
• Start with a quick recap of the material; optionally, show one relevant slide from the modules.
• Involve students in solving problems. Ask them for ideas, and gently explain whether they would work.
• Try to make it like you're solving the problem with them instead of teaching them.
• Don't be too formal (i.e. don't be like an instructor).
• Remind them there are no foolish questions; we're all learning together.
  • Even if they don't know how to solve the entire question, they can suggest the next thing to try.
• If a student suggests a solution that you're unsure about, let them know that you can't tell off the top of your head and will check afterwards.
• It can be helpful to go through a practice tutorial beforehand with an invisible audience to get the right words.

Programming Questions on Marmoset

Auto-testing is done on Marmoset (potentially with additional handmarking). Final results are uploaded to MarkUs so students have all the information in one place.

Setup

A couple of weeks before a programming question is released to students, ask instructors for a draft and start thinking of test cases and writing a solution. (If this is a programming question that was used in previous terms, the archives may have some tests or solutions already written.)

Come up with an outline for the tests you want to include and check with the instructor to verify it covers everything important. Tests should be comprehensive, including edge cases and checks for efficiency and memory leaks. However, tests should also not overlap significantly. There can be a few combined tests, but there should also be tests for partial marks.

Detailed documentation on setting up Marmoset using Marisa can be found here: https://maria-marisa.notion.site/. You can also look in ~/archives/<term>/marmoset for previous setups using Marisa.

Things to watch out for:

• Marmoset only supports up to 20 marks for an assignment. Thus, there can be at most 20 tests worth marks, as each test must be worth a non-negative integer number of marks.
• Marmoset has a default time limit of 10s and an output file memory limit of 4MB. These can be changed in lib/marmoset_lib (for all tests) or in test.properties (for individual tests).
• Make sure that any testing around whitespace is intended.
  • To ignore trailing whitespace on lines, add -Z to the files_match command in test/default_template. files_match is simply a wrapper for diff; other options can be passed as well.

Note that AID dropboxes don't use Marisa. Just make a copy of a previous P01-AID folder (which can be found in archives). The folder name, file in sol/, and one of the scripts in test/ that checks the filename need to be modified. The test doesn't check the contents, only the filename; the AIDs are checked separately (see Academic Integrity Declaration Check above).

Downloading marks and submissions

This is done using the ~/bin/marm_sql script in the course account. Note that all commands include a 10-minute grace period after the deadline (discuss with instructors if this is still wanted).

Terminology: "valid users" are users that are either in the classlist, or are specified as exceptional cases (students in ~/.exceptionlist.YYMM). INCs or other "not officially enrolled" students who still need to submit to Marmoset and be graded should generally be added to the .exceptionlist file, so that their marks are not ignored by this script.

The -v option used in the commands below is not essential. It enables "verbose mode", which for these commands causes a progress bar to appear.

The commands below assume submissions and marks are being downloaded for Programming Question 1, with project name P1 on Marmoset and short identifier P1 on MarkUs.

1. Download the best ontime submissions (the project names are the ones shown on Marmoset): ~/bin/marm_sql -v -d P1
   This will create a folder in the current directory called P1. Under this directory, there will be folders for each valid user who submitted to the project. Note this means that if a folder does not get created for a valid user, this means they did NOT submit anything (this is probably good to know for AID checking).
2. Do the same for AID submissions: ~/bin/marm_sql -v -d P01-AID
3. Create the assignment separately in Markus. Don't allow web submission, since we are only using Marmoset submissions.
4. Add the marking criteria for the assignment. Usually there is one criterion, "Correctness", to which the Marmoset grade is assigned to.
5. Create repositories for students on Markus: /u/isg/bin/markus.py create_group PQ1
6. Check in student Marmoset submissions to student Markus repos: /u/isg/bin/markus.py upload_svn PQ1 P1
7. Do the same for AID submissions: /u/isg/bin/markus.py upload_svn PQ1 P01-AID
8. Collect all assignments in markus (even late, since we only uploaded ontime ones).
9. Grades should be in a nested folder, ~/handin/PQ-Grades. Change into that folder and download the best ontime marks: ~/bin/marm_sql -v -m P1
   This will create a file in the current directory called project-ProjectName-grades.csv. The file format is just a series of lines of the form "userid,mark" for each valid user. If a valid user did not submit anything, they still appear in the file but their mark is recorded as 0.
10. Convert the marmoset csv into a format that Markus scripts can use and provide the criteria name: ./marm_to_markus_csv.sh project-P1-grades.csv "Correctness" >> PQ1_markus.csv
11. Upload marks from the csv file to the Markus assignment: /u/isg/bin/markus.py set_marks_csv PQ1 PQ1_markus.csv
    This will automatically fill in the correctness mark for all assignments.

Note that students that drop the course will still have a group created for them in step 5, although they will not have any submissions/marks downloaded from Marmoset. Their mark needs to be manually set to 0 before all assignments can be marked as complete and released to students. (There might be a better way, but there aren't usually that many students that drop.)

Email Guidelines

• Do not answer any student questions or give away any info if it is not sent from their uwaterloo.ca email. Instead, tell them to email from their uwaterloo account. If the student email from their friendly email account, then they need to include their Quest username in the body of the email because students may share names, etc.
• Always CC the CS 240 course email to keep a record of all emails. In case you forget and realise later, just forward your reply to the CS 240 email. Every ISA then gets a notification that it is being dealt with. It also creates an audit trail if a student files a grievance and we have to provide evidence of our interactions with the student in a complete thread.
• Always sign off on the email and include "CS240 ISA". There are many people on the team and we need to know who is responding. Also for students, it is nice having a name attached to the email.
• Use the name students have signed off with on emails when addressing them.
• Make sure to always respond from the course email account, and not your personal email account.
• If an email is 'Read' but not resolved then flag it so it isn't missed. Conversely, if an email is flagged but you replied to the student and the situation is resolved, then you can unflag it.
• If you are unsure how to reply to something always ask the relevant party who knows how to. Do not assume.
• If there are any administrative emails like missing assignments, assignment exemptions, assignment extensions, midterms, midterms exemptions, cheating cases, VIFs etc, forward them to Karen to bring to her notice. CC the forward to the course account so that others can easily see it is done.
• In the case of replying to emails with sensitive matters or inflammatory comments/impressions,
  • decide if this is a matter for Karen (ISC) to handle (and let Karen know) or
  • if another ISA should check the reply email before sending to make sure it follows basic principles of the Waterloo workplace.
• If an email looks unreplied to and is not flagged, then make sure to search in all emails and check in case it was indeed resolved. Sometimes the subject line changes start a new email thread and we don't want to email the student multiple times.

TODO:

• Tutorial question archive on the course account, categorized by topic/module and including tex sources and pdfs with the question, ideas for how to present it in the tutorial and a sample solution.

Key Links

• Examination Duties - Everything you need to prepare for exams.
• Office Tips
• MarkUs
• CourseWebpage
• PiazzaTips -- Q/A forum
• Autotesting Procedure
• Distributed RST

Non-Technical (To be moved to D2L)

Outdated

• https://cs.uwaterloo.ca/twiki/view/ISG/MarkUsGroupsSVNRepos#Auto_testing_Procedure

RST Public tests

• files can be found in ~/marking/A#/test.pt The main file to edit is runTests, which you need to ensure is updated before the assignment is made public on Markus. It checks student submissions and verifies that the required files are included. Make sure that the file names and the term name are up to date.
• For assignments with programming questions the public test also runs a basic test of their code. This is generally a test using sample input/output that was included with the question.

RST Programming assignments:

• Normal procedure is the same as normal assignments: The ISA needs to create a marking scheme and hold the marking meeting if there is hand-marking component.
• The MarkUs setting depends on if this is an individual assignment or an embedded question in a normal assignment
• Normally, the ISA needs to come up with all test cases and set up each test case in private tests. You can follow the previous terms' format to make this or use a script called make-in. To use the make-in script, You need to first copy-paste the provided_files folder from some previous terms' folders, and change the number of folder and the .in , .out files accordingly. Note here you'll need to update the computeMarks-postprocess and the iotest accordingly (test name and test program name)
• To assign more than 1 mark to one test cases, rename the folder that you need to change under the ./in/7b/testx_1, change the 1 here to the value you want
• After all submissions are collected and all test cases set up, you can run RST on this assignment. Make sure you check the option: "Let students run and view basic/public test results (requires RST/Bittersuite)". The detailed procedure can be seen at RST page, or use this command:

rst -t t -s 'a*' aXX suitename testrunname

Note here "a*" is the assignment name. suitename is the number/character you use to identify the test suite, normally it is just "1". testrunname can be anything.

• After you finish running the test, follow this website to upload marks: https://cs.uwaterloo.ca/twiki/view/ISG/MarkUsScripts#set_marks_rst%20sub-command. Use this command to add the OUTPUT.txt file to the students MarkUs directory: (Here I use the AUTOTESTRESULTS as the testrunname)

  /u/isg/bin/markus.py upload_marking --prefix=testresults --filter=OUTPUT.txt AX /u/cs240/marking/test.1.AUTOTESTRESULTS

• If your marking scheme has a different weighting for the programming question than the number of test cases you have, use the conversion script:

~/bin/rst2csv ~/marking/PQX_autotest/test.1.AUTOTESTRESULTS ~/PQX_marks_combined.csv WW

• Where X is the programming question number such as 3 and WW is the weight of that programming question such as 12.
• After generating the CSV file use the following command to upload the marks to MarkUs:

~isg/bin/markus.py set_marks_csv PQX ~/PQX_marks_combined.csv

• Where X is the programming question number such as 3
• When everything finishes and post-mortem is ready, make a public note on Piazza with brief information for each test case indicating what they are.
• To get some stats on which tests everybody failed, use the stats script:

~/bin/stats PQX_autotest

• Where 'PQX_autotest' is the _autotest folder for the PQ you want stats on, such as 'PQ1_autotest'

Testing Student code for Assignments with Programming Questions

• MarkUs does not do any autotesting. There are a few examples from previous terms for public test of file names and possibly the programming questions.
• An automatically run script places all student submissions from MarkUs to handin folder of course account.
• For assignments with programming, you must run a bash script to test the student submissions in handin, then record the marks for every student individually within their folders before reuploading all the marks to students' MarkUs. See Markus Prepare For Marking for details.

<!-- <h6>Testing Submissions</h6> <p>There is a script in the course account called ~/bin/test-student which can be used for testing student submissions. running it without arguments will give a help message describing how to use it.</p> <p>There are also similar scripts test-parallel, test-upload, test-stats which run test-student in parallel, upload autotest results to markus and calculate mark distribution over the tests respectively.</p> -->

No Longer Relevant

-- PatrickLee - 18 Aug 2011

Academic Integrity Declaration