Marmoset Testing Script Guide

In the cs246t account, each project is usually organized as

~
  bin/                        -- this is where all the scripts are
  marmoset/
    termID/                   -- like F12, or S12 
      bin/                    -- this is where the runtest script is. Copy it from a previous term
      a1/
        a1pX/
          sol/                -- this is usually where you would put the solution to the problem
          sol_wrong/          -- this is usually where you would put a wrong solution, to make sure your test doesn't pass everyone
          test/               -- this is where you would make the tests
            code/             -- in this directory has all the testing scripts
              solution/       -- in this directory has the solution for testing purposes, you can just copy from ../../../sol/
              marm_lib.bash   -- has a lot of routines for testing
              template        -- use it to create a generic script that works for all tests
              maketests       -- use it to create the individual test scripts from template
            TestMakeFile      -- preprocessing of student code to make sure it satisfies project requirements
            test.properties   -- Here are the test names as well as their category
            testlist          -- contains a list of all testnames
        a1pY/
          etc..
      a2/
        etc...

Template Test Script

An example template is shown below: (with some complicated question specific testing parts removed)

     1   #!/bin/bash
     2   source marm_lib.bash
     3   TESTNAME=`basename $0`
     4   
     5   #ulimit -f 10
     6   
     7   check_timeout() {
     8      if [ $RVAL = 129 ]; then
     9         RVAL=$TIMEOUT
    10      elif [ $RVAL = 153 ]; then
    11         RVAL=$TIMEOUT
    12      else
    13         RVAL=$CORRECT
    14      fi
    15   
    16      return $RVAL
    17   }
    18   
    19   #test for a6 part4
    20   
    21   OPTIONS=`cat $TESTNAME.in`
    22   
    42   timeout -t 5s -- ./solution/sol_p4 $OPTIONS < HumanPlayer.input 1> $TESTNAME.expected 2> $TESTNAME.expectederr
    43   
    44   timeout -t 5s -- ./Hearts $OPTIONS < HumanPlayer.input 1> $TESTNAME.out 2> $TESTNAME.err
    45   RVAL=$?
    46   RESULT=$RVAL
    47   timeout -t 5s -- valgrind ./Hearts $OPTIONS < HumanPlayer.input 1> $TESTNAME.valout 2> $TESTNAME.valerr
    48   timeout -t 5s -- valgrind_parse $TESTNAME.valerr 1> $TESTNAME.parseout 2> $TESTNAME.parseerr
    49   MEMLEAK=$?
    50   cat $TESTNAME.parseout
    51   check_timeout
    52    
    75   retrn $RVAL &&
    76   retrn $MEMLEAK &&
    77   marm_diff $TESTNAME.out $TESTNAME.expected &&
    78   should_not_contain ".\+" $TESTNAME.err &&
    79   true

The basic structure of the template is: source the marm_lib.bash file (which you should copy from a previous test, and should be in the current directory) get the test name (store it in TESTNAME), so you can create unique output, and expected output files for each test. Have a check timeout routine to check for timeouts.

You may need to compile their code here if you need to (for example: you are testing student's code with your own driver). Run their code using timeout (Note: You need to redirect both output and error or else timeout won't work as you expect)

What timeout does:

timeout -t 5s -- command-to-run-program

will terminate their program if it takes longer than 5s. You can change the time limit by changing 5s to 10s and so forth.

After running their code, store the exit value in a variable (in this case its RVAL), and run the check_timeout routine. Run the solution, so you generate the expected output.

In this case, I ran valgrind to check their code for memory leaks as well. valgrind_parse exists in ~/bin

   retrn $RVAL &&
   retrn $MEMLEAK &&
   marm_diff $TESTNAME.out $TESTNAME.expected &&
   should_contain ".\+" $TESTNAME.err &&
   true

This checks to see if the output is the same as the expected (marm_diff, in marm_lib.bash), their error file is non-empty (should_contain, in marm_lib.bash), if their code timed out (which is stored in RVAL), and if they had memory leaks (which is stored in MEMLEAK).

maketests

     1   #!/bin/bash
     2   
     3   
     4   for i in `cat testlist`; do
     5       echo "$i:"
     6       cp template ${i}
     7       touch $i.in
     8       
     9   done

This takes every file name in testlist and creates a copy of template named that, as well as creating a (empty) input file. You can modify this depending on the order that you create your tests. If you created the input files first, then you would use those to create copies of the template script and add them to the testlist like:

#!/bin/bash
for i in *.in; do
echo "${i/.in/}:"
cp template ${i/.in/}          #create copies of the template
echo ${i/.in/} >> testlist     #add it to the testlist
done

Testmakefile

     1   TEST_SUBMIT_FILES =Makefile
     2    
     3   all:
     4      @excludefile Makefile Hearts
     5      @checkfiles ${TEST_SUBMIT_FILES}
     6      @c++_security_check
     7      @no_c_functions
     8      @cp -r code/* ./
     9      @chmod u+x solution/sol_p4
    10      #$(MAKE) 
    11      @chmod u+x `cat testlist`

TEST_SUBMIT_FILES should be the files that they are to submit for the project

Several commands that are useful:

• excludefile only one of the files in the arguments can be submitted.
• checkfiles all of the files in the arguments must be submitted (-o flag = at least one of the files must be submitted)
• checksecurity no submitted files are allowed to have the commands rmdir, rm, or chmod. (for bash programs. use security_check to only check for chmod)
• c++_security_check no submitted files are allowed to have the function "system()".
• no_c_functions no submitted files are allowed to use c-style I/O and memory management.

Things that needs to be done:

• cp -r code/* ./ -- copy things in code to the current directory
• chmod u+x .... -- give x permission to executable files
• chmod u+x `cat testlist` -- make the test scripts executable

Tips

• Use timeout on every command that you think might even have the tiniest possibility of going into an infinite loop.
• Remember to redirect both the output and error of every call to timeout.
• marm_lib.bash can be found in all previous years' tests
• Read marm_lib.bash and familiarize yourself with all the functions in it
• look at previous years' tests for examples on how to do things
• If you ever need to retest students' submissions on marmoset by uploading a new test suite, DO NOT upload any other tests, or else marmoset will bug out, drop a bunch of submissions and have a bunch of weird errors

Improvements (suggested changes, once you make them, remove from this page)

• put check_timeout in marm_lib.bash
• put marm_lib.bash in ~/bin and source it from there instead of copying it everywhere
• use egrep in marm_lib.bash's should_contain and should_not_contain routines
• write a script to check that students' code contain the line #!/bin/bash (so they can't submit compiled c++ code to the bash section)

-- KaiyuWu - 11 Dec 2012