DryDock Installation Guide

Deepak Giridharagopal
Principal Developer

Updated: 2003/11/03 20:05:56
Version: 1.16

Abstract

This guide shows you how to install DryDock and get it up and running.


1. First steps

Thanks for trying our DryDock! Right off the bat, it should be noted that DryDock interacts with some fairly complicated external systems. The goal of DryDock's installation system is to insulate the user from these complexities as much as possible, but you should know your way around a UNIX system and be comfortable with installing and configuring Apache and MySQL. Chances are, however, if you're looking at deploying DryDock, you already possess the necessary skills. So let's get it on with the installin'!

2. Required software

Python 

Python is an interpreted, interactive, object-oriented programming language, and it's DryDock's native tongue. DryDock requires at least Python version 2.2. If you're running Linux and using a distribution like RedHat or Gentoo, then Python should already be installed. Otherwise, you can see if your distro/vendor has a package available for download, or you can just install from the sources on the Python website.

Important: If you compile Python from sources, you MUST configure it with the --with-threads flag. Both DryDock and the Webware application server it runs atop make heavy use of threading. DryDock's configure script will check for this.

Once you've got Python up and running, you can run DryDock's configure script. The script should tell you what you still need to install to get things working. This what the output should look like if everything checks out (which it probably won't unless you're a DryDock samurai):

Code listing 2.1: Running the DryDock configure script

// Move to the directory you expanded the DryDock archive to
# cd /home/foo/DryDock

// Running configure.py will check for dependencies
# python configure.py
Checking Python version... 2.2.3 (#1, Jul 16 2003, 18:21:23) 
[GCC 3.2.2]
Checking for Python threading support... yes
Checking for MySQL DBI libraries... import failed
No module named MySQLdb

You need to install the Python MySQL DBI module:
http://sourceforge.net/projects/mysql-python

Don't panic about the failure. At this point, you should have passed at least the Python version check and the Python threading check.

MySQL 

MySQL is an open-source, relational database. DryDock should be able to coexist with version 3.x or 4.x of MySQL, but it has only been thoroughly tested with the 4.x series. Binaries for multiple platforms are available at the MySQL website, and your distro/vendor probably has an even more convenient way to get it onto your machine. While installing MySQL is outside the scope of this document, there are a few notable configuration options that DryDock requires.

First, DryDock requires InnoDB support. InnoDB is a MySQL table type that features (among other niceties) full support for transactions. While InnoDB support exists for MySQL 3.x, we've never tried it. Newer releases of MySQL, however, should include InnoDB support out-of-the-box.

Second, DryDock uses some long-running connections to the database in its connection pool. If one of these connections hasn't been used for a bit, the MySQL server closes them and DryDock will complain the next time it tries to use that connection. Though we've written DryDock to automatically recover from such an error, it would probably be a good idea to increase the connection timeout. You do this by editing a variable in your my.cnf file:

Code listing 2.2: In your my.cnf file

read_buffer_size = 2M
myisam_sort_buffer_size = 64M
thread_cache = 8
query_cache_size = 32M
## Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8

set-variable=thread_stack=512k
// Change the connection timeout value to, say, 24 hours
set-variable=wait_timeout=86400

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows 
# (via the "enable-named-pipe" option) will render mysqld useless!
# 
#skip-networking

DryDock has no trouble working with an instance of MySQL running on a remote host. However, DryDock's installation program assumes that there is a user root that can create new databases and tables. If your database is on a remote machine, you'll need to configure MySQL to allow admin access from a root user from the machine that DryDock is on.

Note: If you are running MySQL on a remote machine, the MySQL client libraries and headers still need to be installed on the machine DryDock will be running on.

Code listing 2.3: Add an admin user from a remote machine

// On the server hosting mysql  
# mysql -u root -p mysql
  
// After you've logged in
mysql> INSERT INTO user VALUES('drydock_host','root',PASSWORD('some_pass'),
    ->        'Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y');
mysql> FLUSH PRIVILEGES;

// Now, on 'drydock_host', this should work:
# mysql -h mysql_host -u root -p mysql
Enter password: some_pass

MySQL-Python libraries 

Since DryDock is written in Python, it requires some Python database access libraries for it to talk to MySQL successfully. You can download the MySQLdb module from Sourceforge.

Code listing 2.4: The DryDock configure script, take 2

// Move to the directory you expanded the DryDock archive to
# cd /home/foo/DryDock
# python configure.py
Checking Python version... 2.2.3 (#1, Jul 16 2003, 18:21:23) 
[GCC 3.2.2]
Checking for Python threading support... yes
Checking for MySQL DBI libraries... 0.9.2
Checking for valid Config.py... import failed
There is no defined value for APPDIR.

Did you edit the Config.py as per the installation guide?

Again, don't get freaked out about the failure. :) But at this point, you should have passed the MySQL DBI check.

CVS 

DryDock uses CVS as its document revisioning system. While your distro/vendor probably has packages for this, if necessary you can download the source from the CVS website. Installation is a straightforward ./configure; make; make install process.

The Webware application server 

DryDock runs on top of Webware, a Python-based web-application suite. Specifically, DryDock uses the Webware application server, WebKit. A source tarball can be obtained from the Webware homepage. Installing Webware is beyond the scope of this document, but the source archive comes with a straightforward _README file. There is also an excellent WebKit Installation Guide located in the archive at WebKit/Docs/InstallGuide.html.

Note: Webware comes with a variety of adapters that let it talk to a web server. If you use Apache, you can use either a CGI-based adapter or a dedicated Apache module. The examples in the rest of this guide assumes that you used the CGI-based option.

Once you've got a copy of WebKit up and running, you need to modify the application server's launch script to point at DryDock's library code. A $PYTHONPATH variable needs to be set prior to the application server's startup:

Code listing 2.5: Editing the WebKit/AppServer script

#!/bin/sh

# as long as the appserver returns a 3, it wants to be restarted

retcode=3
while test $retcode -eq 3; do
    // Add the following line pointing $PYTHONPATH to DryDock
    export PYTHONPATH=/path/to/DryDock:$PYTHONPATH
    /usr/bin/env python Launch.py NewThreadedAppServer $*
    retcode=$?
done

Now we need to tell WebKit where DryDock web-based frontend is located. We do this by adding DryDock to WebKit's list of contexts in WebKit/Configs/Application.config:

Code listing 2.6: Editing the WebKit/Configs/Application.config file

'ActivityLogFilename':  'Logs/Activity.csv',
'ActivityLogColumns':   ['request.remoteAddress', 'request.method', 'request.uri', 
                         'response.size', 'servlet.name', 'request.timeStamp', 
                         'transaction.duration', 'transaction.errorOccurred'],

'Contexts': {
               'Admin':         '%(WebKitPath)s/Admin',
               'Examples':      '%(WebKitPath)s/Examples',
               'Testing':       '%(WebKitPath)s/Testing',
               'default':       '%(WebKitPath)s/Examples', ##MAWD
	       // We need to add DryDock to the list
	       'DD':            '/path/to/DryDock/WebApp',
},

'SessionStore':         'Dynamic', # can be File or Dynamic or Memory
'SessionTimeout':              60, # minutes 

Important: The /WebApp that trails the path to your DryDock installation directory is important, so be sure it's there!

Warning: Though you can technically name the DryDock context whatever you'd like, DO NOT name it DryDock. Doing so may cause some namespace problems when you're running the program. We like DD, ourselves. :)

And that should do it as far as Webware goes. At this point, you're now ready to edit DryDock's own configuration files. This shouldn't take nearly as long as what you've done so far.

3. Configuring DryDock

Editing the Config.py file 

The main DryDock configuration file is located at DryDock/Config/Config.py; you should give this file a good read through. The defaults should be okay for most setups, but there are a few settings that require customizing.

Here is an example where the required settings are filled in for a fictitious site:

Code listing 3.1: Example configuration values

// Close to the top of the file...
# Where DryDock is installed
#
# Example: APPDIR = "/home/foo/DryDock_app"
APPDIR = "/opt/DryDock"

# The directory where all of the editable web content is stored. This should
# be the directory that users will be uploading their documents for approval.
#
# Example: LOCALDIR = "/home/foo/web_root"
LOCALDIR = "/usr/share/htdocs"

# The DryDock "working" directory. This will contain files that are created by
# DryDock during its use. Note that when you run the configure script, the
# directory contained in this value will be blown away and re-created. FYI.
#
# NOTE: Changing this variable after you've done a "make install" won't work
# immediately. You have to stop Webware, move your old working directory to the
# new location, change that WORKING_DIR variable to point at the new location,
# and then restart.
#
# Example: WORKING_DIR = "/home/foo/DryDock_data"
WORKING_DIR = "/var/opt/DryDock_data"

# What user and group your instance of WebWare is running as. This should be in
# standard UNIX colon-seperated chmod notation.
#
# This variable is used during the "make install" process. If you change this
# value, be sure to chown the DryDock source tree and the working directory so
# that the specified user:group can access the files.
#
# Example: WEBWARE_IDENTITY = "webware:webware"
WEBWARE_IDENTITY = "webware:webware"

# Where Webware is located. This directory should contain WebKit, among other
# Webware components
#
# NOTE: Changing this variable after you've done a "make install" won't work
# immediately. You need to stop Webware, move it to the new location, change
# this variable, and then restart.
#
# Example: WEBWARE_LOCATION = "/home/foo/Webware"
WEBWARE_LOCATION = "/opt/Webware"

# What is the prefix to the DryDock web app as configured in WebWare? This is
# the URL relative to the root of your web server that you would need to type
# to get to DryDock through WebWare. MUST INCLUDE TRAILING "/"!
#
# Example: If you configured Webware to house DryDock at context /DD and you
# use the Webware CGI adapter located at '/cgi-bin/WebKit.cgi', then you
# would use:
#
# HREF_PREPEND = "/cgi-bin/WebKit.cgi/DD/"
HREF_PREPEND = "/cgi-bin/WebKit.cgi/DD/"

// In the 'CVS Options' section...
# The location of the CVS executable
# 
# Example: CVS_LOCATION = "/usr/bin/cvs"
CVS_LOCATION = "/usr/bin/cvs"

// In the 'Email Options' section...

// The email address needs to be a tuple. The first item is
// the barebones email address, and the second item is the
// email address the way it should be shown in the email's
// headers.
# The from address to use. This cannot be a list of
# email addresses, it must be a single address!
SMTP_FROM_ADDRESS = ('deepak@arlut.utexas.edu', 'DryDock <deepak@arlut.utexas.edu>')

Warning: You should also look at the values for DB_USERNAME and DB_PASSWORD. It's probably a good idea to change them from the default. The password is (unfortunately) stored in plaintext right now, but the file should only be readable by the user Webware runs as.

Note: Once you've done a "make install" (at the end of this document), there are certain Config settings that will be difficult to change. The comments in the Config file should tell you which variables can't easily be altered after DryDock is installed.

Once you've editited the configuration file to your satisfaction, you can check and see if it parses correctly by running the configure.py script again. If you're missing a required variable, then the script will let you know:

Code listing 3.2: The DryDock configure.py script, take 3

# cd /path/to/DryDock
# python configure.py
Checking Python version... 2.2.3 (#1, Jul 16 2003, 18:21:23) 
[GCC 3.2.2]
Checking for Python threading support... yes
Checking for MySQL DBI libraries... 0.9.2
Checking for valid Config.py... looks good
Checking for valid COLUMNS file... looks good
Checking for properly installed SyncKit... not found.
No module named SyncKit.SyncMain

You need to choose an appropriate SyncKit before DryDock can be used.
Please refer to the installation guide.

Don't panic if configure gives you an error when it checks for a valid SyncKit. You haven't done that yet. Just make sure that the check for a valid Config.py file passes.

Editing the columns definition file 

Whenever a user approves a file for publication, DryDock can be configured to ask for additional information about the approval. You could ask the user for the name of their supervisor, or a one-line summary of the published file, etc. You tell DryDock what additional information you'd like to collect through editing DryDock's COLUMNS file.

The COLUMNS file is a sequential list of column descriptions. They're called column descriptions because each extra attribute you want DryDock to capture requires an additional column be added to a table in the database. Yeah, the name sucks. :) The COLUMNS file is located at DryDock/Config/COLUMNS. By default, no additional columns are defined. However, there are some example column definitions:

Code listing 3.3

# This is the COLUMNS definition file
# 
# Here are some sample column definitions
#
# [approved_by]
# long name: Approved for public release by
# db type: TEXT
# description: The name of the director or sponsor who gave official approval for this file.
# html input type: TEXT
# visibility: SIGNFIELD

# [restrictions]
# long name: Restriction type
# db type: TEXT
# description: What type of access control is being used to protect the file(s) in question.
# html input type: COMBO{NONE:No protection,PASSWORD:Password protected}
# visibility: REVIEWFIELD_MANDATORY

Here is a breakdown of the particulars:

The last 2 fields require a bit more explanation.

The html input type determines what kind of form element is displayed for the user to input this column's data. Right now, your only choices are TEXT or COMBO. TEXT fields will simply take input from a standard textbox. COMBO, however, will build an HTML "select" object that displays a dropdown box.

The COMBO format looks strange, but it's not too bad to figure out. It goes something like: COMBO{value:text, value:text, value:text...}. The value is what the form element will return when the form is submitted. The text is what the user actually sees. The following code snippet shows an example:

Code listing 3.4: Example COMBO defition

// A COMBO definition of this...
[college]
...
html input type = COMBO{Harvard:You are a Harvard monkey,Yale:You are a Yale parrot}
  
// ...will product HTML that looks something like this...  
<SELECT NAME="college">
  <OPTION VALUE="Harvard">You are a Harvard monkey
  <OPTION VALUE="Yale">You are a Yale parrot
</SELECT>

Warning: Do NOT put spaces between value and text fields, or before or after the commas that seperate them. This will cause the parser to fail.

The last field is visibility. This can take on one of 3 values.

And that's all there is to the COLUMNS file. Hopefully you're ready to edit it if necessary. You can re-run the configure script to ensure that your COLUMNS file changes (if you've made any) are legal. You should get the same output as the previous running of configure.py.

Note: Make sure that you've got your column definitions the way you want them! Once you do a "make install" (at the end of this document), there are certain settings you can't easily change without much labour. You won't easily be able to add or remove columns or change their db type without afterwords doing another "make install" (which will re-initialize the database, among other things)!

Setting up a SyncKit 

The last thing to do is configure a SyncKit. DryDock's main purpose is to synchronize it's tree of approved content with some other machine or apache root. This way, developmental content is cleanly seperated from production content. The problem is that designing an all-purpose sync kit is a pain - everyone's network environment is different (often radically so). Our solution was to just make it easy to customize DryDock's sync functionality for your environment.

DryDock ships with a kit called SimpleSyncKit. It simply takes all of the files approved by DryDock and copies them to a directory called live_web inside your DryDock working directory. The kit also makes sure to backup a previous live_web directory in the process (the backup is in old_web). By configuring an apache instance to serve content from live_web, it will always be serving up current approved content. The follwing code snippet shows how to do this:

Code listing 3.5: Configuring Apache to point at the live_web directory

// In your apache.conf (or http.conf) file

// Replace "path_of_your_choosing" with your desired URL base,
// and replace "WORKING_DIR" with the path to your DryDock
// working directory as specified in Config.py.
Alias /path_of_your_choosing /WORKING_DIR/live_web

Since it's a simple kit that will work for anyone, we recommend you start off using it. For more details on how to customize SyncKits, refer to the SyncKit Guide. For now, though, we'll just use the SimpleSyncKit:

Code listing 3.6: Setting up a SyncKit

// Change to the DryDock Config directory. The "path_to_DryDock" should
// be the same one you defined in the APPDIR variable in the Config.py file
# cd /path_to_DryDock/DryDock/Config

// Create a symbolic link to the kit you will use
# ln -s SimpleSyncKit SyncKit

One mo' time 

Now you can re-run configure.py (hopefully for the last time). At this point, everything should check out. And if it doesn't, the configure.py script should give you some indication as to what went wrong. If you think you've done everything right and the configure script is insane, don't hesitate to drop a line to the DryDock users mailing list.

Code listing 3.7: The DryDock configure.py script, the last take

# cd /path/to/DryDock
# python configure.py
Checking Python version... 2.2.3 (#1, Jul 16 2003, 18:21:23) 
[GCC 3.2.2]
Checking for Python threading support... yes
Checking for MySQL DBI libraries... 0.9.2
Checking for valid Config.py... looks good
Checking for valid COLUMNS file... looks good
Checking for properly installed SyncKit... looks good
Checking for CVS executable... /usr/bin/cvs
Checking for DryDock context definition in Webware... found

Everything looks good...now do a 'make' followed by a 'make install'.

4. Building and installing

Make 

Once you've managed to successfully run configure.py, you can go ahead and run make. The make process will:

Make install 

If the make was sucessful, you can now do a make install. This will:

And then you should be ready to start up DryDock!

Note: You need to make sure that the authenticator you're using can recognize the admin user you've designated. For example, if you are using the StandardUNIXAuthenticator, you need to make sure that the admin user is defined as a real user on this system.

5. Running DryDock

Starting the application server 

Now you're ready to start the application server and start messing around with DryDock's web interface. To start the application server:

Code listing 5.1: Starting Webware

// Replace "path_to_Webware" with the location you installed Webware to.  
// This is the same path you will have used as "WEBWARE_LOCATION" in Config.py.
# cd /path_to_Webware/WebKit

// Now start the ball rolling...
# ./AppServer

And that should do it! You can now go to the DryDock login page by visiting http://localhost/HREF_PREPEND/, replacing HREF_PREPEND with the value of that variable as defined in your Config.py file. You should get a page that looks something like:


Figure 5.1: A sample DryDock login page


Fig. 1: DryDock login page

6. Last steps

Now that you've (hopefully) gotten everything working, you can start messing around with DryDock's functionality. Before you do, you might want to give the Quickstart Guide a read through. It will tell you about DryDock's major features and how to use them. I know you're a computer wizard that doesn't need any stinking instructions, but the guide is light on text and big on screenshots. And who doesn't like screenshots?

If you're having difficulty getting things going, or to talk about anything DryDock-related, please don't hesitate to email the DryDock users mailing list. I'm on there, and I'm a swell dude who's more than willing to help you out. :) Happy motoring!



Outline

1. First steps

2. Required software
- Python
- MySQL
- MySQL-Python libraries
- CVS
- The Webware application server

3. Configuring DryDock
- Editing the Config.py file
- Editing the columns definition file
- Setting up a SyncKit
- One mo' time

4. Building and installing
- Make
- Make install

5. Running DryDock
- Starting the application server

6. Last steps

Links

1. Python website
2. DryDock's configure script
3. MySQL website
4. download the MySQLdb module from Sourceforge
5. CVS website
6. Webware homepage
7. the previous running of configure.py
8. SyncKit Guide
9. Quickstart Guide

Valid XHTML 1.0!

Valid CSS!