Testing Guide

If this is your first time reading through the testing guide, please read the full summary below.

Quick Links

CorrectnessTests - information on creating different tests for questions and general rules to follow.

options.rkt - information on local and global options settings for correctness tests.

BasicTests - creating basic tests for questions.

BanningFunctions - understanding how to restrict function usage for questions.

StepperQuestions - setting up testing for stepper questions.

ConstantsAndHelpers - setting up constants and helpers to be used to assist in creating test cases.

CheckingHelpersRecursion - checking if students are using local/global helper functions and which ones they are using, checking if functions are recursive or have recursive helpers, etc

ProvidingLibraries - using created constants and helpers in your testing.

ProofreadingTestSetups - tips to check setups before running tests on students' submissions.

SpecialCases/CommonIssues - miscellaneous helpful topics and general issues that should be kept in mind.

OnlineExams - setting up online midterms/finals.

File Layout

Brief File/Directory Descriptions

Most of what you need is under the marking directory in the course account. From here you can go into a specific assignment. The convention is to label the assignment directories aXX, where the XX is the number of the assignment. Under each assignment directory should be a test.0 and a test.pt directory.

There may also be some test.0.something and test.pt.something directories. The "something" can be any string, and is just used as a label. These directories contains test results for students. One level down will be directories labeled with student's quest ids. These contain the results for that individual student.

Descriptions in Point form:

1. test.pt - This directory contains all basic tests setups. Basic tests are created before the assignment is released to students. These are used to help students catch simple errors.

2. test.0 - Similar to test.pt, test.0 contains all correctness tests setups. These tests are used to fully test and grade a student's submission. All setups in test.0 need to be finished before the assignment is due (or at last collected and tested).

3. Computemarks (click on this link for more information) - This is the initial script that is run when we talk about "running tests" through rst. bittersuite (/in tests) are run from here. This should not need any modification.

4. options.rkt (click on this link for more information) This tells the script which language to use, which file to load, etc.

5. Computemarks-postprocess - This is the last script that RST runs after running bittersuite (the in/ directory tests). The filenames at the top of computemarks-postprocess file need to be updated for every assignment. Note that if students submit too fast while basic tests are running, computemarks-postprocess complains and lets them know that tests were interrupted.

6. test.rkt (click on this link for more information) - This is the place where put the actual tests.

7. provided (click on this link for more information) - This is the place where all the scripts and helper files we use go.

8. config.ss (click on this link for more information) - This file specifies options that will apply to this entire BitterSuite run.

9. in - This contains directories for every question on the assignment and is where the tests go.

---------------------------------------------------------------------------------------------------------------------------------------

Some Information in more detail:

One level into test.0 there is a provided and an in directory, as well as some other files. The header file contains the text that will show up at the top of all test results for the assignment. Other files not listed in the diagram should not need any modification.

Regarding provided, for basic details on including libraries see this section link. For more in-depth details, or if you are running into errors, see this section link.

Regarding "in", within each question directory will be directories starting at 001 going up to the last test that you want to write for a question. These directories contain a test.rkt file which is an individual test for that question. Basic tests will usually only need just one test, so there will only be a 001 for that question. Stepper questions have a different format, see this section link for more information.

Computemarks

• Before it even begins running tests, there are a few checks that are done first. The script will first check to make sure that the student has completed a00, which is the initial set up assignment that all students are required to complete. It also checks to make sure that all the files the student has submitted are written in plaintext, so as to avoid issues with students using gracket (students uploading racket files with some sort of graphical item included). The last check is to make sure that the student has files uploaded to MarkUs in the first place.
• You may need to turn of a00 checking for online exams.
• You may need to update gracket note with the right link.
• There may be other things you want to change.

More details on this script can be found on the MarkUsScripts page

test.rkt

The format of these files in basic/correctness tests is very simple, the contents have the following form. Note that equality-operators are functions like equal? and =.

(result (equality-operator (student-function our-test-input) expected-output))
(expected true/false)      

;; You can also have format like the following. However the first one is prefered because it avoids revealing our test(s).

(result (student-function our-test-input))
(expected expected-output)

The expression in the result line will be evaluated in the language chosen in the options file. The value produced will then be compared to the value in the expected line, using the function equal?. If these two values are the same, and there were no errors while evaluating any expressions, the submission will pass the test. Unfortunately, this is not a built-in form in Racket, so you can only check them in check-expect form or use rst to actually run them on the server. You can still call other Racket functions in the test case, even something like a cond with multiple cases depending on what the student's function does. There will be more tips on writing tests in the basic and correctness test sections.

Note that for steppers, test.rkt is slightly different. You can refer to this StepperQuestions for more details.

Also note that you are not limited to only use expected and result in test.rkt. You can refer to this page BitterSuiteScheme for more information.

Coming Up With Tests

More information on BasicTests and CorrectnessTests can be found on their respective pages.

Additional Information

In CS135 it is common practice to ban certain built-in functions from being used, more information on how to do this can be found on the BanningFunctions page

When setting up tests, sometimes issues may come up, information on these problems can be found on the SpecialCases/CommonIssues page

Once you have completed a test setup for an assignment, it is very important that you look over your work to make sure that it has been done correctly, and that there will be no issues once students start submitting. For more information on how you can look over your work, you should check out the ProofreadingTestSetups page