MarkUs Installation Issues

See also: MarkUs

Modified Files

The following files have been modified in our local installation:

• markus/app/views/main/login.html.erb to account for our login method
• markus/config/database.yml
• markus/config/environment.rb
• markus/config/environments/development.rb
• markus/config/environments/production.rb
• markus/config/mongrel_cluster.yml (probably only differs in the test instance)
• markus/lib/repo/subversion_repository.rb (a hook for updating our testing system)

Modifying Files

If you modify a file, chances are pretty good that you need to stop and restart the mongrel cluster to get the changes to show up:

cd markus
mongrel_rails cluster::stop
mongrel_rails cluster::start

Reset Installation

To reset the installation to a pristine state (ie, drop all the tables and recreate them). By default, the following affects the development or test environment. If you really, really want it to work on the production environment (do you really want that?) you need to set the RAILS_ENV to something. Research it on the MarkUs web site.

• mongrel_rails cluster::stop
• rake repos:drop # destroys repositories at REPOSITORY_STORAGE
• rake db:reset # resets db and reloads schema
• rake markus:instructor first_name="Byron Weber" last_name="Becker" user_name="bwbecker"
• mongrel_rails cluster::start

Logs

Log from installing v 0.6.1

• Untarred the new version into it's own directory.
• Compared and then copied markus/config/database.yml from old to new
• Compared and then copied markus/config/environment.rb from old to new
• Compared and then copied markus/config/environments/production.rb from old to new
• Compared and then copied markus/lib/repo/subversion_repository.rb from old to new
• Copied markus/app/views/main/login.html.erb from old to new
• Fired it up using the test URL (but the live database and repository)
  • NOTE: I don't know that this is safe. Do as little as possible while there are two systems running against the same database and repository. It's just to make sure there isn't a surprise.
  • Stopped the test environment's mongrel cluster
  • Relinked markus-test from the real test environment to the new environment for testing
    • rm markus-tst
    • ln -s markus-0.6.1 markus-test
  • Modified config/mongrel_cluster to use the production environment and port 8003
  • started the cluster for the new environment
  • Played around a bit. All looks good.
  • Shut down the mongrel cluster
• Fired up as production
  • Changed the mongrel_cluster file for port 8000
  • Reset the markus-test symlink
  • Stopped the production cluster
  • Changed the name of the new installation's directory to markus-prod-0.6.1
  • Pointed the markus symlink to the new directory
  • Restarted the cluster

Log from installing v 0.7.0

• I want to try to isolate things more by course and term. Therefore, created a new directory, cs115-s10.
• Downloaded and untarred v0.7.0 into cs115-s10.
• Compared, copied, edited config/database.yml. Changed name of database to markus_cs115_s10_prod.
• Compared, copied, edited config/environment.rb
  • Commented out RAILS_GEM_VERSION (it was commented out in copied version)
  • Changed the session_key from "_markus_session_prod" to "_markus_session_cs115_s10_prod"
  • Changed the secret
• Edited config/environments/production.rb (COURSE_NAME, REPOSITORY_STORAGE, added POST_COMMIT_HOOK) Need to create the directory for the REPOSITORY_STORAGE.
• Created the directory cs115-s10/uw and copied the post-commit program into it.
• Compared lib/repo/subversion_repository.rb. There were some changes, so copied the post-commit hook code from old to new.
• Compared app/views/main/login.html.erb to original in v0.6.1; no changes so copied over Omar's replacement.
• linked the markus sym-link: ln -s cs115-s10/markus-0.7.0 markus
• Run following commands:
  • RAILS_ENV="production"
  • export RAILS_ENV
  • cd ~/cs115_s10/markus-0.7.0/
  • rake db:create (gave error: undefined method '[]' for nil:NilClass)
    • Mike Conley says rake has a bug that ignores the RAILS_ENV variable for creating the database.
    • In config/database.yml, temporarily changed "production:" to "development:".
    • Backed up config/environments/development.rb. Copied config/environments/production.rb to development.rb.
    • rake db:create
    • rake db:schema:load
    • rake markus:instructor first_name='Byron Weber' last_name='Becker' user_name='bwbecker'
    • Changed config/database.yml back to "production:" and removed config/environments/development.rb
  • mongrel_rails cluster::start
• Seems to be up and running.

Note for multiple instances: the file config/mongrel_cluster.yml will need to be modified to refer to different port numbers. These will generally ascend by 3: one instance will have port: "8000", another port: "8003", and so on.

-- ByronWeberBecker - 26 Feb 2010

Log from attempt at installing v 0.9.0 on www.student.cs

The new version is now being installed on services024.student.cs instead of on scspc137.cs. Also referring to MarkUs Version >= 0.6 Deployment Documentation (A System Administrator's Guide) for reference.

• This will eventually be for two courses, but is currently for testing only. Therefore, created a new directory, f10-dev.
• Downloaded and untarred v0.9.0 into f10-dev; cd f10-dev.
• svnadmin create svn-markus-install-repos; svn import markus-0.9.0 file://$(absolute svn-markus-install-repos); rm -r markus-0.9.0; svn checkout file://$(absolute svn-markus-install-repos) markus-0.9.0
• As instructed in INSTALL file in unarchived markus directory,
  • export RAILS_ENV="production"
  • comment out the line containing RAILS_GEM_VERSION="2.3.8" in config/environment.rb
  • cp config/database.yml.postgresql config/database.yml
• Did NOT make any of the following changes to config/environment.rb as suggested above; there is no longer a key and secret, and there doesn't seem to be a need to change config.load_paths:
  • Changed the value of key from _markus_session to _markus_session_f10-dev
  • Changed the secret
  • Did NOT change config.load_paths; this is the same as 0.7.0 but with an extra element in the list, and there may be a good reason for that.
• Edited config/environments/production.rb
  • COURSE_NAME = "CS Fall 2010 Development"
  • REPOSITORY_STORAGE REPOSITORY_STORAGE = "/u/markus/f10-dev/svn-repos-root". Created the directory for the REPOSITORY_STORAGE as 0.7.0 notes suggest this is necessary.
  • POST_COMMIT_HOOK = "/u/markus/f10-dev/uw/post-commit". Added to the bottom of the file; this is to be used below.
  • PDF_STORAGE = "/u/markus/f10-dev/converted_pdf_dir/". Created this directory to stop the installer's complaints.
  • Changed all remaining instances of /home/markus to /u/markus to match UW filesystem conventions.
• Created the directory f10-dev/uw and copied a slightly modified post-commit program into it.
• Leaving the repository hook for now, but this will need to be resolved.

[markus@services024]:markus-0.9.0$ diff -wbBi lib/repo/subversion_repository.rb <(ssh markus@scspc137 -- cat /home/markus/cs115-s10/markus-0.7.0/lib/repo/subversion_repository.rb)                                        
2d1
< require "svn/client"
31,34d30
<   if !defined? CLOSEABLE_VERSION
<     CLOSEABLE_VERSION = "1.6.5"
<   end
< 
50d45
<     @closed = false
72,77c67
<     repository = Svn::Repos.create(connect_string, {}, fs_config) #raises exception if not successful
< 
<     if SubversionRepository.closeable?
<       repository.close
<     end
< 
---
>     Svn::Repos.create(connect_string, {}, fs_config) #raises exception if not successful
87,154d76
<   # Static method: Yields an existing Subversion repository and closes it afterwards
<   def self.access(connect_string)
<     repository = self.open(connect_string)
<     yield repository
<     repository.close
<   end
< 
<   # Static method: Deletes an existing Subversion repository
<   def self.delete(repo_path)
<     Svn::Repos::delete(repo_path)
<   end
< 
<   # method : Export an existing Subversion repository to a new folder
<   def export(repo_dest_dir, revision_number=nil)
<     # Modify the path of the repository
<     # If libsvn-ruby raise a segfault, check the first argument of
<     # Svn::Client::export which must be an URI (ex : file:///home/...)
< 
<     repo_path_dir = "file://" + expand_path(@repos_path)
<     ctx = Svn::Client::Context.new
< 
<     # don't fail on non CA signed ssl server
<     ctx.add_ssl_server_trust_file_provider
<     setup_auth_baton(ctx.auth_baton)
<     ctx.add_username_provider
< 
<     # username and password
<     ctx.add_simple_prompt_provider(0) do |cred, realm, username, may_save|
<       cred.username = "markus"
<       cred.password = "markus"
<       cred.may_save = false
<     end
< 
<     # Raise an error if the destination repository already exists
<     if (File.exists?(repo_dest_dir))
<       raise(ExportRepositoryAlreadyExists, "Exported repository already exists")
<     end
< 
<     begin
<       result = ctx.export( repo_path_dir, repo_dest_dir, revision_number, nil)
<     end
< 
<     return result
<   end
< 
<   # Static method:  Returns whether or not the available Svn library supports
<   # closing
<   def self.closeable?
<     return Svn::Client.version.to_s >= CLOSEABLE_VERSION
<   end
< 
<   # Closes the repository
<   def close
<     if self.class.closeable?
<       @repos.close
<     end
<     @closed = true
<   end
< 
<   # Returns whether or not repository is closed
<   def closed?
<     if self.class.closeable?
<       return @repos.closed?
<     end
<     return @closed
<   end
< 
< 
269a192,207
>     
>     #
>     # Invoke the post-commit hook
>     # Added by bwbecker 091219
>     
>     
>     pipe = IO.popen(POST_COMMIT_HOOK, "w+")
>     pipe.puts("transaction.user_id=#{transaction.user_id}")
>     pipe.puts("repository_path=#{@repos_path}")
>     jobs.each do |job|
>       pipe.puts("filepath=#{job[:path]}")
>     end
>     pipe.close
> 
>     # End additions
> 
491,499d428
<   #Converts a pathname to an absolute pathname
<   def expand_path(file_name, dir_string = "/")
<     expanded = File.expand_path(file_name, dir_string) 
<     if RUBY_PLATFORM =~ /(:?mswin|mingw)/ #only if the platform is Windows
<       expanded = expanded[2..-1]#remove the drive letter
<     end
<     return expanded
<   end
<   
698,703d626
<   # Function necessary for exporting the svn repository
<   def setup_auth_baton(auth_baton)
<     auth_baton[Svn::Core::AUTH_PARAM_CONFIG_DIR] = nil
<     auth_baton[Svn::Core::AUTH_PARAM_DEFAULT_USERNAME] = nil
<   end
< 
793c716
<     path = expand_path(path, "/")
---
>     path = File.expand_path(path, "/")
[markus@services024]:markus-0.9.0$ 

• Not copying app/views/main/login.html.erb yet; it's unclear what impact the REMOTE_USER reading capabilities will have on this section.

See MarkUsSetupFall2010FullLog for more details of setup.

TODO: Once modifications are confirmed working, the post-commit utility likely should be uploaded to this page for easy copying in the future

Log from installation of v 0.9.0 on markus002.student.cs

• This will eventually be for two courses, but is currently for testing only. Therefore, created a new directory, f10-dev.
• Downloaded and untarred v0.9.0 into f10-dev; cd f10-dev; tar xzf ../markus-0.9.0.tar.gz
• ... and so on as with the www.student.cs notes above. Currently, stopped around the same spot.

Specific files

post-commit

This should be in ~/$course-$term/uw on the markus account; it is used to synchronize the MarkUs repository with the handin directory on the course account. It also creates a log in the same directory of the actions it's taken.

The current version as of Oct 5, 2010:

#!/usr/bin/python

import sys, re, subprocess, time

# The following two constants must be updated with course/term-specific information
course_name = 'cs135'
term_identifier = 'f10'

log = open('/home/markus/%s-%s/uw/post-commit.log' % (course_name, term_identifier), 'a')


# Taken from student.cs:/u/isg/bin/rst-files/distributed-rst/common.py
def print_msg(msg,f=log):
   if f is None: f=sys.stdout
   print >>f, "%s: %s"%(time.ctime(), msg)
   f.flush()


print_msg('Starting')

print_msg('Standard input contents:')
lines = sys.stdin.readlines()
for line in lines:
   log.write(line)
log.flush()

log.write('\n')
print_msg('Parsed results:')
(tag, username) = lines[0].strip().split('=')
log.write(tag + ":" + username + '\n')
assert tag == 'transaction.user_id'
(tag, repo_path) = lines[1].strip().split('=')
log.write(tag + ":" + repo_path + '\n')
assert tag == 'repository_path'
(tag, path) = lines[2].strip().split('=')
log.write(tag + ":" + path + '\n')
assert tag == 'filepath'
assignDir = re.match('[^/]*', path).group()
log.write(assignDir + '\n')


updateCmd = "ssh %s@linux.student.cs.uwaterloo.ca 'cd /u/%s/markus/bin/; " % (course_name, course_name) + \
   "./markusSubmission.py %s %s %s'" % (repo_path, username, assignDir)
   
log.write('\n')
print_msg(updateCmd)
#subprocess.call(updateCmd, shell=True)
data = subprocess.Popen(updateCmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True).stdout.read()
print(data)
log.write(data)

updateCmd = "ssh %s@linux.student.cs.uwaterloo.ca -- " % (course_name) + \
   "'/u/isg/bin/submit/submit_pub_test_hook %s %s %s'" % (course_name, assignDir, username)
log.write('\n')
print_msg(updateCmd)
data = subprocess.Popen(updateCmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True).stdout.read()
print(data)
log.write(data)


print_msg('All done!\n\n')
log.close()

Improvements to Consider

One clear improvement to post-commit would be to remove the need for the course_name and term_identifier constants. The first use for them is in the construction of a path to the log file; this suggests strongly that they can/should also be read directly from the path to the executable itself.