Local Marmoset Submission Mirror

The goal of the scripts here is to be able to create a local cache of Marmoset submissions on the course account (typically in the handin directory) so operations can be done on them more conveniently.

Getting the most recent of the highest-marked submissions for each student

All of the following scripts should be in the same directory. Two files must be downloaded manually from Marmoset until a stable way to download them from the command-line has been implemented and documented properly: the zip file of all submissions made by all students, and the record of all grades in CSV format.

prepare_submissions

The entry point; takes too many parameters, which could all be derived from the assignment name given Marmoset's standard naming convention.

#!/usr/bin/env bash

# Should be replaced with a cron utility that downloads the zip file
# into a subdirectory of /tmp and then makes this call to 
# marmoset_acquire_directories

export PATH=$(/bin/showpath gnu standard)

if [[ $# -ne 3 ]]; then
    echo "Usage: $(basename "$0") <zip-file> <markfile> <assignment_name>" >&2
    exit 1
fi

tmpdir="/tmp/.$(whoami).marmoprep.$$"
mkdir "$tmpdir"
cp "$1" "$tmpdir"

pushd "$tmpdir" >/dev/null 2>&1
unzip "$(basename "$1")"
rm "$(basename "$1")"
popd >/dev/null 2>&1

"$(dirname "$0")/marmoset_acquire_directories" "$tmpdir" "/u/$(whoami)/handin/$3" "/u/$(whoami)/course/ta_marmoset/$3/submissionlogs" < "$2"
rm -r "$tmpdir"

marmoset_acquire_directories

#!/usr/bin/env bash

# Given a mark file as input and a directory containing subdirectories of all
# submissions for all students, copy the appropriate renamed subdirectories 
# to the destination directory.
# Write the version copied to the destination directory into the log directory.

if [[ $# -ne 3 ]]; then
   echo "Usage: $(basename $0) <marmoset_files_directory> <destination_directory> <most_recent_version_logdir>" >&2
   exit 1
fi

readonly orig_dir="$1"
readonly dest_dir="$2"
readonly logs_dir="$3"

if [ ! -d "$orig_dir" ]; then
   echo "Directory containing marmoset files ($orig_dir) cannot be found" >&2
   exit 2
fi

# By convention, lock around the submission cache directory
# while doing any activity that needs synchronization.
# Since flock requires this to be a file, not a directory,
# create a dummy file in it; only do this if one doesn't already
# exist to avoid modifying anything about the file.
mkdir -p "$dest_dir"
globlock="$dest_dir/.marmolockfile"
if [ ! -e "$globlock" ]; then
   touch "$globlock"
fi

mkdir -p "$logs_dir"

# Copy the most recent file versions.
# Echo versioning information into the log directory.
for submissionid in $($(dirname "$0")/marmoset_choose_recent_best); do
   # TODO: Do this extraction better.
   userid=$(echo "$submissionid" | cut -f1 -d-)

   lockpid=$(flock "$globlock")
   if [[ -e "$logs_dir/$userid" && $(cat "$logs_dir/$userid") -eq "$submissionid" ]]; then
      echo "$userid is up to date as $submissionid"
   else
      echo "Copying $submissionid to update submission for $userid"
      full_destdir="$dest_dir/$userid"

      mkdir -p "$full_destdir"
      rm "$full_destdir"/*
      cp "$orig_dir/$submissionid"/* "$full_destdir"
      echo "$submissionid" > "$logs_dir/$userid"
   fi
   kill $lockpid
done

marmoset_choose_recent_best

#!/usr/bin/env bash

# Consumes as input the "Student Grades" file for ALL submissions.
# Returns the directory name corresponding to the most recent
# best submission.

readonly tmpfile="/tmp/marmoset_choose_recent_best_tmp1.$$"

export PATH=$(/bin/showpath -PackageWarnings gnu standard)

touch "$tmpfile"
chmod 600 "$tmpfile"

# Get rid of the useless first line
read

# Now sort by userid and dump the result into a temporary file.
sort > "$tmpfile"

# TODO: Prepend an identifier instead of appending because Marmoset
# sometimes leaves trailing mystery marks on the grade sheet.
# Then increment the values below, retest to make sure it still works.

for userid in $(cut -f1 -d, < "$tmpfile" | uniq); do
   # Grab this userid from the cache file; apply ordering numbers.
   # Then reverse sort by grade to prioritize high marks; reverse sort by UTC
   # stamp secondarily to priorititze *recent* marks; then grab the first line
   # to get the most recent good mark.
   # Save that submissionid.

   submissionid=$(fgrep "$userid" < "$tmpfile" | "$(dirname $0)/marmoset_number_names" | sort -n -r -t , -k 4,4 -k 3,3 | head -n 1 | cut -f1 -d,)
   echo $submissionid
done

rm "$tmpfile"

exit 0

marmoset_number_names

#!/usr/bin/env perl

# Number userids so they match up with Marmoset's directory naming convention.

use strict;
use warnings;

my $count = 1;

while (<>) {
   my ($userid, @restofline) = split(',');
   print ("$userid-" . $count++ . ',' . join(',', @restofline));
}

find_nonrepeats

#!/usr/bin/env bash

# Find students who did not resubmit after being marked once already.

if [[ $# -ne 1 ]]; then
   echo "Usage: $(basename $0) assign" >&2
   exit 1
fi

assign="$1"
readonly basedir="/u/$(whoami)/course/ta_marmoset/$assign"

verifydir () {
   if [[ ! -d "$1" ]]; then
      echo "Log directory $1 for assignment does not exist" >&2
      exit 5
   fi
}

readonly sublog="$basedir/submissionlogs"
readonly marklog="$basedir/markingcache"

verifydir "$basedir"
verifydir "$sublog"
verifydir "$marklog"

readonly tmpdir="/tmp/.$(whoami).findnon.$$"
mkdir -p "$tmpdir"
chmod 700 "$tmpdir"

readonly file1="$tmpdir/sublog"
readonly file2="$tmpdir/marklog"

# Newest submissionID goes into file1
cat "$sublog"/* > "$file1"

# Marked submissionIDs go into file2
pushd "$marklog" >/dev/null 2>&1

for file in *; do
   echo "$file" | cut -f2 -d_
done > "$file2"

popd >/dev/null 2>&1

# Find all repeated submission IDs.
# Then, print only these.
cat "$file1" "$file2" | sort | uniq -c | perl -ne 'if (/^\s*2\s*([^-]*)-/) { print "$1\n"; }'

rm -r "$tmpdir"

Other Method?

Grab info from PrintingMarmosetAssignments.