Conditions for passing/failing correctness tests and check test cases

Terminology Clarifications:

• Syntax Error: Failing check test case is not a syntax error. For example:

      (define (foo x) (first x))
      (check-expect (foo empty) "incorrect")
      ;; Is a failing test case, but not a syntax error, there are no errors when this is run, only the ?Test Results? window complains in Dr Racket

• Check Test Cases: This includes check-expect, check-within, check-error, etc

Basic/Correctness Tests

Note that basic tests & correctness tests do not run student check expect/within/error etc. All code outside of check test cases is run for every basic/correctness test. A correctness test can fail if:

• There is a syntax error in the file
• There is a runtime error (like infinite recursion for example) in the code that run outside of check test cases
• The required function is incorrect in come way
• If a disallowed function is used anywhere in the file, even if it is not used in the part being tested by that correctness test. (Instructors also note while designing assignments, the permitted functions list must be the same for the whole file)
• If A0 is not complete (this setting can be change in the “computeMarks” file of a test setup)

Note if a check test case runs forever, fails because of function not defined, or fails for another reason, this doesn’t affect basic/correctness results. So, there is no reason to not write the rest of the design recipe even if you haven’t written the function definition. Of course, if the syntax itself for a check test case is wrong, i.e. something like wrong number of parameters passed in the check-expect then basic/correctness fails.

Check Test Cases

Students can lose check test case marks if:

• They are missing required check test cases
• There is a syntax error in the file when it is run, including running check test cases (Note that a failing check test case is not a syntax error)
• There is a runtime error once the file is run, including running check test cases:
  • Wrong usage of functions with respect to language level being used
  • Or something like using (append …) with only one parameter in beginner student levels where it is wrong (in intermediate student it works), there are other functions as well that can behave differently based on language level

Students won’t lose marks just because of:

• Using disallowed functions in their file which are defined appropriately in that language level – only basic/correctness tests care about this
• Not completing a00
• The check test case fails for their code, for example:
• The function being tested is not defined
• The produced and expected value do not match
• If the arguments of function call inside the check test case do not follow the contract for the tested function – this just means this function will not pass the check test case validity tests, and this check test case will not be counted. If all their other check test cases include all the required check test cases, then they can still get full marks. Basically – any invalid check test case calls are just ignored but NOT given negative marks for. Only the valid check test cases are considered.

Special Cases:

Students can still get marks if they have more or less arguments than needed in a check test case even though it is a syntax error. If the function call is the first or second argument of the check test case, it will be collected and sent through our checks. If there is no function call, then the check test case is just ignored without any errors. for example:

(define (directed->undirected G)
      (cond [(empty? G) true]))

;; directed->undirected function call args are collected in the following cases
(check-expect (directed->undirected empty) 1 1 1 1)  
(check-expect (directed->undirected empty))  
(check-expect 1 (directed->undirected empty)) ; this is actually not a syntax error

;; This is ignored by the scripts and doesn't cause problems
(check-expect 2)  

-- Bani Singh - 2022-01-04