Course Evaluation instructions for end-users

These instructions will be emailed to end-users of the data. End-users are Math administrators such as Chairs of Math Departments and the CS Director, and CS Director Undergrad and Associate Director Undergrad. Clients have (so far) been fairly clear in a desire for software that doesn't need much documentation.

There are two sets of instructions:

• for administrative end-users who access the data from the command-line (members of unix group cs-eval);
• for administrative end-users who are emailed the data once per term.

In addition, there are instructions sent to regular instructors, via the "evaluate.uwaterloo.ca" system which generates the data. These instructions are archived here as well, though they are not generated by CS/Math.

Email Instructions for command-line users

Web interface

There is now a web-based interface for the course evaluation data, at: https://evaluate.uwaterloo.ca/

It has the same back-end data as the command-line tools, with more detail explicitly explained at the cost of more pointing-and-clicking. Currently, web access for CS faculty is limited to Computer Science and Software Engineering. Software Engineering data includes Math evaluations and Engineering evaluations- which use different templates and cover non-Math Faculty/courses. In contrast, command-line access is for Math faculty evaluations, which includes Math faculty evaluations in Software Engineering.

The web UI should be fairly straight-forward; if there are any questions you can refer to: https://cs.uwaterloo.ca/cscf/teaching/evaluate/administrators.shtml

Command-line interface

The Course Evaluation results ("Tompa Scores") are available via the command-line.

Using the interface:

1. log in to linux.cs.uwaterloo.ca and add an alias that will execute ~cs-eval/evals/report.py
2. run the command report.py with various options

Examples

report.py --term W2012-F2013 - returns all undergrad evaluations for Winter 2012 to Fall 2013

report.py --term 2010.1 --subject "CS|SE" - returns CS and SE evaluations for Winter 2010

report.py --userid jsmith --tag grad - returns all grad courses taught by userid jsmith

report.py --userid jsmith - returns all undergrad courses taught by userid jsmith

CSV Output

The CSV output columns are: 'course_id', 'term_id', 'instructor_name', 'userid', 'course_label', 'section', 'nresp', 'rresp', 'percent_classes', 'hrs_wk', 'prep', 'delivery', 'effect', 'summary', 'course_avg_prep', 'course_avg_delivery', 'course_avg_effect', 'course_avg_summary', 'course_count', 'prof_avg_prep', 'prof_avg_delivery', 'prof_avg_effect', 'prof_avg_summary', 'prof_count', 'tag'

• course_id is a unique identifier matching the same course over all terms regardless of the cross-listings, as determined by the maintainers of Quest data.
• term_id is returned in the Quest format. The first digit is 0 for years < 2000 and 1 for years > 2000. The next two digits are the last two digits of the year. The last digit is the month.
• Columns with _avg_ are over the previous 15 terms (5 years).

Notes on limitations

• The command-line tool has undergrad data going back to Fall 2001. Grad data goes back to 2009.

Help page

This help page may be viewed by running report.py -h

usage: report.py [-h] [-v] [--precision {1,2,3,4,5}]
                 [--tag {grad,all,not-grad}] [--term TERM]
                 [--avg_terms [AVG_TERMS]] [--userid USERID] [--out OUT]
                 [--subject SUBJECT] [--top_instructors]

Report evaluation data for a given term, userid, name, and/or subject.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         display verbose output
  --precision {1,2,3,4,5}
                        precision of results (1-5, default 2)
  --tag {grad,all,not-grad}
                        tag to identify graduate or online data. Accepts a
                        single value or negated value. Default 'not-grad' for
                        not-grad courses. Note that online tag is not
                        currently used.

Action:
  Supply one or more of the following required actions:

  --term TERM           term number (eg., S2010 or 2010.2 or 2010.5 or 1105
                        for May 2010) or range (eg., S2009-F2012)
  --avg_terms [AVG_TERMS]
                        the number of terms to calculate the avg scores. The
                        highest number of terms would be 15 (5 years). The
                        Default is 15. (eg., 1,2,5,7,14)
  --userid USERID       instructor userid
  --out OUT             the format of the output. Printed in eather CSV or
                        Table. default csv
  --subject SUBJECT     course subject (eg., 'AMATH|CS|MATH') - regex format
                        accepted; special parsing so 'MATH' does not include
                        AMATH/PMATH and 'CO' does not include COMM
  --top_instructors     Generate a report for top instructors with 70%
                        response or 25 replies, and all ratings at least 4.1

If you have any questions, please feel free to drop me an email.

Email Instructions for emailed users

The following was emailed to Math Chairs who are emailed the results once a term.

---

The Course Evaluation results ("Tompa Scores") will be emailed shortly; as mentioned previously, they will be in CSV format, with a descriptive header as follows:

Improvements to note:

• input data is normalized against the campus Quest database, as a second source besides hand-entered MUO data. We have an audit trail and a unique course_id regardless of whether the course changed numbers over time.
• Fall 2013 first-year courses have been evaluated online instead of on paper; for future use, we have a tag field to note these (and any additional courses) as 'online' such that we can track whether their evaluations are different compared to paper.

The CSV output columns are: 'course_id', 'term_id', 'instructor_name', 'userid', 'course_label', 'nresp', 'rresp', 'percent_classes', 'hrs_wk', 'prep', 'delivery', 'effect', 'summary', 'course_avg_prep', 'course_avg_delivery', 'course_avg_effect', 'course_avg_summary', 'course_count', 'prof_avg_prep', 'prof_avg_delivery', 'prof_avg_effect', 'prof_avg_summary', 'prof_count', 'tag'

• course_id is a unique identifier matching the same course over all terms regardless of the cross-listings, as determined by the maintainers of Quest data.
• term_id is returned in the Quest format. The first digit is 0 for years < 2000 and 1 for years > 2000. The next two digits are the last two digits of the year. The last digit is the month.
• Columns with _avg_ are over the previous 15 terms (5 years).

Questions? Feel free to drop me an email.

Evaluate emailed instructions to instructors

Sample 1: Spring 2017 (Verbose)

---------- Forwarded message ---------
From: evaluate@uwaterloo.ca <evaluate@uwaterloo.ca>
Date: Tue, Jul 11, 2017 at 8:19 AM
Subject: You have upcoming online student evaluations
To: instructor <instructor@uwaterloo.ca>

You have upcoming student evaluations on evaluate

Evaluate, the online course evaluation system, was developed on campus in order to meet the specific needs of the UWaterloo community. It allows students to complete surveys with any internet-enabled device both in and out of class and greatly simplifies the administrative process.

Which of my courses are being evaluated?

Our records indicate that you are the instructor for the course section(s) listed below, which have been set up for evaluation in Evaluate. Each enrolled student will have access to the course evaluation between the dates indicated below. If a course you are teaching is not listed, or a course is listed that shouldn't be, please email Mirko Vucicevich and your Associate Dean of Undergraduate Studies (or Teaching) to have the issue rectified.

(Note: if a course listed has more than one crosslisted course, only one title will appear below)

Surveys for CS 100 will be open from Sunday July 09 (23:59) to Tuesday July 25 (23:59)

What do I need to do?

Please review the general best practices described below to maximize course evaluation participation.

In advance: Prepare your students

Tell your students in advance which day you plan to give class time for course evaluations and ask them to bring a laptop, smartphone, or tablet with them to class that day. A review of past term course evaluation data shows that the majority of students who complete a given course evaluation do so during the time given in class.

Note: for online courses, students will automatically receive an email asking them to complete the related course evaluation; however, a message directly from you is likely to increase participation.

On evaluation day: Administer the evaluation

On the board at the front of the room, write your name and the course code to help prevent any confusion. Also write the Evaluate web address: https://evaluate.uwaterloo.ca (or use the PowerPoint slide that some Faculties are providing).

Tell your students that their feedback has impact. The best way to convey the value you place on their feedback is by giving an example of a change you’ve made to the course or your teaching because of past student feedback. If there is a specific course element you’d like to receive feedback on (such as a new approach, activity, or assessment), now is the time to ask. Direct students to use an open-ended text question to respond to your prompt. This gives you targeted feedback and can also contribute to higher response rates.

Give time at the beginning or middle of class to maximize response rates. If you wait until the end of class, students may rush through the evaluation or simply pack up and leave.

Let students know that when they log in, they will see a list of all available course evaluations for the current term; ask them to use this time complete the evaluation for your course.

Remind them that the results are completely anonymous; the log in is used only to ensure the correct students access the correct evaluations.

If they have questions, students can contact the developer, whose email address is on the front page of the website.

After evaluation day: Check participation

When you log in to https://evaluate.uwaterloo.ca, you will be able to see what percentage of students completed the course evaluation when you gave class time. If it’s high, thank your students for their participation. If it’s low, consider talking to your students again about the importance of the course evaluation process and ask them to complete the course evaluation before the closing date.

After evaluations have closed: Access your results

Sometime after the end of the "Grades due" period (see https://uwaterloo.ca/quest/undergraduate-students/important-dates), you will be able to review and export your numerical scores and written comments by logging into https://evaluate.uwaterloo.ca.

Feel free to contact Mirko Vucicevich with any questions or comments.

Sample 2: Fall 2017 (terse)

---------- Forwarded message ---------
From: Science Alert <scialert@uwaterloo.ca>
Date: Wed, Jul 4, 2018 at 7:00 AM
Subject: You have online surveys configured in evaluate
To: instructor <instructor@uwaterloo.ca>

Course evaluations have been configured for your classes.

The following courses are now available to be viewed at evaluate.uwaterloo.ca and will be open to students within the next week. Please ensure your students are given in-class time to complete them. If the course is an online course, please be sure to notify students on LEARN when surveys open..

CS 136 section xxx [ Mon, Jul 9 midnight to Thu, Jul 26 midnight ] (prepared by lbell at uwaterloo.ca) CS 136 section xxx [ Mon, Jul 9 midnight to Thu, Jul 26 midnight ] (prepared by lbell at uwaterloo.ca) If you have any questions about any of the evaluations listed above, the administration procedures, or other details, please contact the person who prepared the evaluation.

You can log in to evaluate.uwaterloo.ca to view the status of your surveys, and view response rates for surveys which are open.

-- DanielAllen - 2014-01-30