public abstract class GanymedeBuilderTask
extends java.lang.Object
implements java.lang.Runnable
This class is designed to be subclassed in order to handle full-state synchronization from the Ganymede server to external directory service targets.
GanymedeBuilderTask is really only intended to support directory
service targets that are 'dump and replace' in nature, like NIS and
classical DNS. The reason for this is that GanymedeBuilderTasks
are executed asynchronously with respect to transaction commits.
By the time a GanymedeBuilderTask subclass runs, the Ganymede
server has forgotten all previous states of the data in the server.
All the GanymedeBuilderTask can do is check to see what kinds of
objects (Users, Groups, Systems, etc.) have changed since the last
time it was run. It can also check to see which fields have been
changed, within those object types. It cannot check to see whether
a particular field in a specific object has changed, however.
Since there's no way for a GanymedeBuilderTask to do a
before-and-after comparison, all it can reliably do is to write out
absolutely everything relevant to its synchronization channel, and
then hand off the build to an external process. If you want to do
a more granular, before-and-after incremental build, you can
instead choose to use the more synchronous SyncRunner class, which uses a standard
XML format to represent changes to external synchronization
processes.
Subclasses of GanymedeBuilderTask need to implement the builderPhase1() and
builderPhase2()
methods. builderPhase1() is run while a DBDumpLock is asserted on the server's
DBStore, guaranteeing a
transaction-consistent database state that can be examined.
builderPhase1() should do whatever is required to examine the
database and to determine whether this builder task needs to carry
out a build. If so, builderPhase1() should write out the data
files that will be needed for builderPhase2() and return true.
When builderPhase1() completes, the dump lock is released. If
builderPhase1() returns true, the builderPhase2() method is then
run. This method is intended to run external scripts (typically
written in Perl or Python) that process the files written out by
the builderPhase1() method.
All subclasses of GanymedeBuilderTask need to be registered in
the Ganymede database via the Task object type.
GanymedeBuilderTasks registered to be run on database commit will
automatically be issued by the GanymedeScheduler when
transactions commit. The GanymedeScheduler is designed to run only
a single instance of a task at a time, waiting to issue any new
execution of the task until the previous execution completes. A
GanymedeBuilderTask doesn't finish executing until its
builderPhase2() method returns. This protects your builderPhase2()
method from having its input data being overwritten by the next
builderPhase1() method writing out new data. It also sets a
minimum build-to-build latency, according to how long your
builderPhase1() and builderPhase2() methods take to complete. No
matter how long builderPhase2() takes, the GanymedeScheduler will
run the builds as fast as it can, back-to-back.
Your builderPhase1() method, however, should execute and complete as fast as possible. The dump lock protecting builderPhase1() prevents any transactions from committing while a builderPhase1() method is executing. Any significant delay in transaction commits may cause noticeable delays for your users. Fortunately, since builderPhase1() implementations just scan the Ganymede in-memory database and write out some text files, this usually doesn't take very long. If you find that your builderPhase1() method is taking too long, you may want to consider splitting your build into multiple builder tasks. The GanymedeBuilderTask is designed so that the server can execute multiple distinct builder tasks concurrently. Splitting your build into multiple pieces that can be run concurrently can also improve your build latency by reducing the amount of work that a given external process has to do when called by builderPhase2().
GanymedeBuilderTask includes a set of helper methods that subclasses can take advantage of in order to facilitate their operation.
baseChanged(short)
method can be used from builderPhase1() to
check to see whether objects of a given type have been changed
since the builder task was last run.getOptionValue(java.lang.String)
method makes it possible for a builder task to retrieve
configuration information from the task object in the Ganymede
database which links the task into the server's scheduling.openOutFile(java.lang.String,
java.lang.String) method not only opens files for writing, it also
takes care to manage the archiving of old versions of the emitted
file. The ganymede.builder.backups property controls
this behavior. It should be set to a path for keeping zipped
copies of previous builder outputs if you want this archiving
feature to be in effect. If it is, whenever files are opened by a
builder task using openOutFile(), previous copies of those files
will be moved to the directory set in
ganymede.builder.backups. The first time after
midnight that openOutFile() is called by a builder task, all of the
files written out the previous day by the builder task will be
collected into a single zip file. This can be handy, but you may
not actually want to have certain files archived. In that case,
you should have your external sync script take care to remove such
files after processing them..enumerateObjects(short)
method can be used by builderPhase1() to get an Enumeration of
objects of a given type to examine for writing.In addition, the GanymedeBuilderTask base class logic is responsible for interfacing with the rest of the Ganymede server to display each builder task's status, both in the admin console and in the client. The little conveyor belt icon in the lower left corner of the Ganymede graphical client is controlled by the action of the GanymedeBuilderTask and SyncRunner objects being run after a transaction commit.
| Modifier and Type | Field and Description |
|---|---|
private static java.util.Hashtable<java.lang.String,java.lang.Integer> |
backupsBusy
This hashtable maps directory paths to an Integer
counting the number of tasks that are currently
copying backup files to it.
|
private static java.lang.String |
basePath |
private static java.lang.String |
currentBackUpDirectory |
private static boolean |
debug |
private static boolean |
firstRun |
private boolean |
forceAllBases
If this flag is true, baseChanged() will always return true,
as a way of forcing consideration of all databases that might
be examined by GanymedeBuilderTask subclasses.
|
private scheduleHandle |
handle
A scheduleHandle that we can use to update the admin consoles as
to our build status.
|
protected java.util.Date |
lastRunTime |
(package private) DBDumpLock |
lock |
private static java.lang.String |
oldBackUpDirectory |
protected java.util.Date |
oldLastRunTime |
private java.util.Vector<java.lang.String> |
optionsCache
A list of options that were defined on this GanymedeBuilderTask.
|
private static int |
phase1Count
Count of the number of builder tasks currently
running in phase 1.
|
private static int |
phase2Count
Count of the number of builder tasks currently
running in phase 1.
|
private static long |
rolloverTime |
private static long |
rollunderTime |
private boolean |
runOnCommit
Will be true if this builder task should be scheduled when a
transaction is committed.
|
(package private) GanymedeSession |
session |
protected Invid |
taskDefObjInvid
Must be protected so subclasses in a different package can
set this.
|
(package private) static TranslationService |
ts
TranslationService object for handling string localization in
the Ganymede server.
|
| Constructor and Description |
|---|
GanymedeBuilderTask() |
| Modifier and Type | Method and Description |
|---|---|
protected boolean |
baseChanged(int baseid)
This method is used by subclasses of GanymedeBuilderTask to
determine whether a particular base has had any modifications
made to it since the last time this builder task was run.
|
protected boolean |
baseChanged(int baseid,
java.util.List fieldIds)
This method is used by subclasses of GanymedeBuilderTask to
determine whether a particular field of a particular base has had
any modifications made to it since the last time this builder
task was run.
|
protected boolean |
baseChanged(short baseid)
This method is used by subclasses of GanymedeBuilderTask to
determine whether a particular base has had any modifications
made to it since the last time this builder task was run.
|
protected boolean |
baseChanged(short baseid,
java.util.List<java.lang.Short> fieldIds)
This method is used by subclasses of GanymedeBuilderTask to
determine whether a particular field of a particular base has had
any modifications made to it since the last time this builder
task was run.
|
abstract boolean |
builderPhase1()
This method is intended to be overridden by subclasses of
GanymedeBuilderTask.
|
abstract boolean |
builderPhase2()
This method is intended to be overridden by subclasses of
GanymedeBuilderTask.
|
private static int |
busyCount(java.lang.String path) |
private static void |
cleanBackupDirectory()
This static method is run before the first time a builder task
writes any file on server start-up.
|
private static void |
decBusy(java.lang.String path) |
(package private) static void |
decPhase1(boolean update) |
(package private) static void |
decPhase2(boolean update) |
protected java.util.Enumeration |
enumerateObjects(int baseid)
This method is used by subclasses of GanymedeBuilderTask to
obtain a list of DBObject references of the requested
type.
|
protected java.util.Enumeration<DBObject> |
enumerateObjects(short baseid)
This method is used by subclasses of GanymedeBuilderTask to
obtain a list of DBObject references of the requested
type.
|
protected Invid |
findLabeledObject(java.lang.String label,
short type)
Finds an invid by title.
|
protected java.lang.String |
getLabel(Invid invid)
This method is used by subclasses of GanymedeBuilderTask to
obtain the label for an object.
|
protected DBObject |
getObject(Invid invid)
This method is used by subclasses of GanymedeBuilderTask to
obtain a reference to a
DBObject
matching a given invid. |
protected java.lang.Iterable<DBObject> |
getObjects(int baseid)
This method is used by subclasses of GanymedeBuilderTask to
obtain a list of DBObject references of the requested
type.
|
protected java.lang.Iterable<DBObject> |
getObjects(short baseid)
This method is used by subclasses of GanymedeBuilderTask to
obtain a list of DBObject references of the requested
type.
|
(package private) java.util.Vector<java.lang.String> |
getOptionStrings()
This method returns the Vector of option strings registered
for this task object in the Ganymede database.
|
protected java.lang.String |
getOptionValue(java.lang.String option)
This method retrieves the value associated with the provided
option name if this builder task was registered with
taskDefObjInvid set by a subclass whose constructor takes an
Invid parameter and which sets taskDefObjInvid in
GanymedeBuilderTask.
|
static int |
getPhase1Count()
This is public for GanymedeSession.openTransaction(), as a
hack to support proper updating of the client's status icon on
client connect.
|
static int |
getPhase2Count()
This is public for GanymedeSession.openTransaction(), as a
hack to support proper updating of the client's status icon on
client connect.
|
private static void |
incBusy(java.lang.String path) |
(package private) static void |
incPhase1(boolean update) |
(package private) static void |
incPhase2(boolean update) |
protected boolean |
isOptionSet(java.lang.String option)
This method looks in the optionStrings field in the task
object associated with this task and determines whether the given
option name is present in the field.
|
private static void |
openBackupDirectory(java.lang.String filename)
This method is called before the server's builder
tasks are run and creates a backup directory for
files to be copied to.
|
protected java.io.PrintWriter |
openOutFile(java.lang.String filename)
This method opens the specified file for writing out a text
stream.
|
protected java.io.PrintWriter |
openOutFile(java.lang.String filename,
java.lang.String taskName)
This method opens the specified file for writing out a text
stream.
|
void |
run()
This method is the main entry point for the GanymedeBuilderTask.
|
void |
run(java.lang.Object[] options)
This method is the main entry point for the GanymedeBuilderTask.
|
void |
runOnCommit(boolean state)
Call this method to control whether or not this builder task
should be run when the Ganymede server commits a transaction.
|
boolean |
runsOnCommit()
Returns true if this builder task should be scheduled when a
transaction commits.
|
void |
setScheduleHandle(scheduleHandle handle)
Method used by the Ganymede scheduler to pass us a handle that we
can use to signal the admin console as to our success or failure.
|
(package private) static void |
updateBuildStatus()
This method is called by the GanymedeBuilderTask base class to
record that the server is processing a build.
|
private static final boolean debug
static final TranslationService ts
private static java.lang.String currentBackUpDirectory
private static java.lang.String oldBackUpDirectory
private static java.util.Hashtable<java.lang.String,java.lang.Integer> backupsBusy
private static java.lang.String basePath
private static long rollunderTime
private static long rolloverTime
private static boolean firstRun
private static int phase1Count
private static int phase2Count
protected java.util.Date lastRunTime
protected java.util.Date oldLastRunTime
GanymedeSession session
DBDumpLock lock
private java.util.Vector<java.lang.String> optionsCache
private boolean forceAllBases
protected Invid taskDefObjInvid
private boolean runOnCommit
private scheduleHandle handle
public void setScheduleHandle(scheduleHandle handle)
public final void run()
run in interface java.lang.Runnablepublic final void run(java.lang.Object[] options)
protected final boolean baseChanged(short baseid)
This method is used by subclasses of GanymedeBuilderTask to
determine whether a particular base has had any modifications
made to it since the last time this builder task was run. This
method works because each GanymedeBuilderTask object keeps a
timestamp which records the last time the builder task ran.
baseChanged() just compares that time stamp against the DBObjectBase.lastChange time stamp
that the DBObjectBase class
maintains at transaction commit. Note that this method will
always return true the first time a particular builder task is
run after the server is started. This means that the first time
a transaction is committed when you start your server, your
builder task will wind up doing a full build.
See also the baseChanged(short,
java.util.List) version of this method, which allows you to
specify a list of fields that you are interested in testing.
baseid - The id number of the base to be checkedprotected final boolean baseChanged(int baseid)
This method is used by subclasses of GanymedeBuilderTask to
determine whether a particular base has had any modifications
made to it since the last time this builder task was run. This
method works because each GanymedeBuilderTask object keeps a
timestamp which records the last time the builder task ran.
baseChanged() just compares that time stamp against the DBObjectBase.lastChange time stamp
that the DBObjectBase class
maintains at transaction commit. Note that this method will
always return true the first time a particular builder task is
run after the server is started. This means that the first time
a transaction is committed when you start your server, your
builder task will wind up doing a full build.
See also the baseChanged(short,
java.util.List) version of this method, which allows you to
specify a list of fields that you are interested in testing.
Note: This variant of baseChanged() takes an int and casts it to a short to remove the need from casting literals on the caller's behalf. If the baseid does not fit in the 16 bit two's complement short range, an IllegalArgumentExceptio will be thrown.
baseid - The id number of the base to be checkedprotected final boolean baseChanged(short baseid,
java.util.List<java.lang.Short> fieldIds)
This method is used by subclasses of GanymedeBuilderTask to
determine whether a particular field of a particular base has had
any modifications made to it since the last time this builder
task was run. This method works because each GanymedeBuilderTask
object keeps a set of timestamp which records the last time the
builder task ran. baseChanged() just compares that time stamp
against the DBObjectBaseField.lastChange time
stamp that the DBObjectBaseField class maintains at
transaction commit. Note that this method will always return
true the first time a particular builder task is run after the
server is started. This means that the first time a transaction
is committed when you start your server, your builder task will
wind up doing a full build.
Otherwise, if none of the fields listed have changed since this GanymedeBuilderTask was last run, baseChanged() will return false. If any of the fields in the fieldIds list have changed since this GanymedeBuilderTask last ran, baseChanged() will return true.
baseid - The id number of the base to be checkedfieldIds - A list of java.lang.Short's containing the
id numbers of the fields to examine.protected final boolean baseChanged(int baseid,
java.util.List fieldIds)
This method is used by subclasses of GanymedeBuilderTask to
determine whether a particular field of a particular base has had
any modifications made to it since the last time this builder
task was run. This method works because each GanymedeBuilderTask
object keeps a set of timestamp which records the last time the
builder task ran. baseChanged() just compares that time stamp
against the DBObjectBaseField.lastChange time
stamp that the DBObjectBaseField class maintains at
transaction commit. Note that this method will always return
true the first time a particular builder task is run after the
server is started. This means that the first time a transaction
is committed when you start your server, your builder task will
wind up doing a full build.
Otherwise, if none of the fields listed have changed since this GanymedeBuilderTask was last run, baseChanged() will return false. If any of the fields in the fieldIds list have changed since this GanymedeBuilderTask last ran, baseChanged() will return true.
Note: This variant of baseChanged() takes an int and casts it to a short to remove the need from casting literals on the caller's behalf. If the baseid does not fit in the 16 bit two's complement short range, an IllegalArgumentException will be thrown.
baseid - The id number of the base to be checkedfieldIds - A list of java.lang.Short's containing the
id numbers of the fields to examine.protected final java.util.Enumeration<DBObject> enumerateObjects(short baseid)
This method is used by subclasses of GanymedeBuilderTask to obtain a list of DBObject references of the requested type.
Note that the Enumeration returned by this method MUST NOT be used after builderPhase1() returns. This Enumeration is only valid while the base in question is locked with the global dumpLock obtained before builderPhase1() is run and which is released after builderPhase1() returns.
baseid - The id number of the base to be listedDBObject referencesprotected final java.util.Enumeration enumerateObjects(int baseid)
This method is used by subclasses of GanymedeBuilderTask to obtain a list of DBObject references of the requested type.
Note that the Enumeration returned by this method MUST NOT be used after builderPhase1() returns. This Enumeration is only valid while the base in question is locked with the global dumpLock obtained before builderPhase1() is run and which is released after builderPhase1() returns.
Note: This variant of enumerateObjects takes an int and casts it to a short to remove the need from casting constants on the caller's behalf. If the baseid does not fit in the 16 bit two's complement short range, an IllegalArgumentException will be thrown.
baseid - The id number of the base to be listedDBObject referencesprotected final java.lang.Iterable<DBObject> getObjects(short baseid)
This method is used by subclasses of GanymedeBuilderTask to obtain a list of DBObject references of the requested type.
Note that the Iterable returned by this method MUST NOT be used after builderPhase1() returns. This Iterable is only valid while the base in question is locked with the global dumpLock obtained before builderPhase1() is run and which is released after builderPhase1() returns.
baseid - The id number of the base to be listedDBObject referencesprotected final java.lang.Iterable<DBObject> getObjects(int baseid)
This method is used by subclasses of GanymedeBuilderTask to obtain a list of DBObject references of the requested type.
Note that the Iterable returned by this method MUST NOT be used after builderPhase1() returns. This Iterable is only valid while the base in question is locked with the global dumpLock obtained before builderPhase1() is run and which is released after builderPhase1() returns.
Note: This variant of getObjects() takes an int and casts it to a short to remove the need from casting constants on the caller's behalf. If the baseid does not fit in the 16 bit two's complement short range, an IllegalArgumentException will be thrown.
baseid - The id number of the base to be listedDBObject referencesprotected final DBObject getObject(Invid invid)
DBObject
matching a given invid.invid - The object id of the object to be viewedprotected final java.lang.String getLabel(Invid invid)
invid - The object id of the object label to be retrievedprotected final Invid findLabeledObject(java.lang.String label, short type)
public abstract boolean builderPhase1()
This method is intended to be overridden by subclasses of GanymedeBuilderTask.
This method runs with a dumpLock obtained for the builder task.
Code run in builderPhase1() can call enumerateObjects() and baseChanged(). Note that the Enumeration of objects returned by enumerateObjects() is only valid and should only be consulted while builderPhase1 is running.. as soon as builderPhase1 returns, the dumpLock used to make the enumerateObjects() call safe to use is relinquished, and any Enumerations obtained will then be unsafe to depend on.
public abstract boolean builderPhase2()
This method is intended to be overridden by subclasses of GanymedeBuilderTask.
This method runs after this task's dumpLock has been relinquished. This method is intended to be used to finish off a build process by running (probably external) code that does not require direct access to the database.
For instance, for an NIS builder task, builderPhase1() would scan the Ganymede object store and write out NIS-compatible source files. builderPhase1() would return, the run() method drops the dump lock so that other transactions can be committed, and then builderPhase2() can be run to turn those on-disk files written by builderPhase1() into NIS maps. This generally involves executing an external Makefile, which can take an indeterminate period of time.
It is important that any external process run by builderPhase2() blocks until it is finished doing the build. If the external script tries to put itself into the background and return early, the Ganymede server will conclude that the external build has completely finished, and it will feel free to immediately schedule this builder task again, which may mean that the builderPhase1() method will overwrite files the backgrounded external builder process is still using.
Note as well that the Ganymede server makes no guarantee as to what the environment variables or current working directory will be set to when any external builder scripts are executed. If your external scripts depend on these things, you should make sure that your external builder script sets them itself.
By releasing the dumpLock before we get to that point, we minimize contention for users of the system.
As a result of having dropped the dumpLock, enumerateObjects() cannot be called by this method.
builderPhase2() can throw a ServiceNotFoundException or ServiceFailedException to indicate to the admin console that the build failed, or it can simply return a boolean result to indicate the same in a less specific fashion.
builderPhase2 is only run if builderPhase1 returns true.
protected boolean isOptionSet(java.lang.String option)
This method looks in the optionStrings field in the task object associated with this task and determines whether the given option name is present in the field. This works only if this builder task was registered with taskDefObjInvid set by a subclass whose constructor takes an Invid parameter and which sets taskDefObjInvid in GanymedeBuilderTask.
That is, if the task object for this task has an option strings vector with the following contents:
useMD5 useShadow
then a call to isOptionSet with 'useMD5' or 'useShadow', of any capitalization, will return true. Any other parameter provided to isOptionSet() will cause false to be returned.
protected java.lang.String getOptionValue(java.lang.String option)
This method retrieves the value associated with the provided option name if this builder task was registered with taskDefObjInvid set by a subclass whose constructor takes an Invid parameter and which sets taskDefObjInvid in GanymedeBuilderTask.
getOptionValue() will search through the option strings for the task object associated with this task and return the substring after the '=' character, if the option name is found on the left.
That is, if the task object for this task has an option strings vector with the following contents:
useMD5 buildPath=/var/ganymede/schema/NT useShadow
then a call to getOptionValue() with 'buildPath', of any capitalization, as the parameter will return '/var/ganymede/schema/NT'.
Any other parameter provided to getOptionValue() will cause null to be returned.
final java.util.Vector<java.lang.String> getOptionStrings()
This method returns the Vector of option strings registered for this task object in the Ganymede database.
If there are no option strings defined, an empty Vector will be returned.
public final boolean runsOnCommit()
public void runOnCommit(boolean state)
protected java.io.PrintWriter openOutFile(java.lang.String filename)
throws java.io.IOException
This method opens the specified file for writing out a text stream.
If the ganymede.builder.backups property is set
to a path in the Ganymede server's ganymede.properties file,
openOutFile() will look to see if the filename provided as a
parameter already exists. If it does, it will be copied to a
subdirectory of the ganymede.builder.backups
directory. This subdirectory will be named with the date in
which the backups therein were copied.
If ganymede.builder.backups is set, the first
time openOutFile() is called after midnight, openOutFile will zip
all the files in any preceding days' backup subdirectories into
one zip file per day.
filename - The fully specified path to the file to openjava.io.IOExceptionprotected java.io.PrintWriter openOutFile(java.lang.String filename,
java.lang.String taskName)
throws java.io.IOException
This method opens the specified file for writing out a text stream.
If the ganymede.builder.backups property is set
to a path in the Ganymede server's ganymede.properties file,
openOutFile() will look to see if the filename provided as a
parameter already exists. If it does, it will be copied to a
subdirectory of the ganymede.builder.backups
directory. This subdirectory will be named with the date in
which the backups therein were copied.
If ganymede.builder.backups is set, the first time
openOutFile() is called after midnight, openOutFile will zip all
the files in any preceding days' backup subdirectories into one
zip file per day.
filename - The name of the file to opentaskName - The name of the builder task that is writing this
file. Used to create a unique name (across tasks) for the backup
copy of the file when we overwrite an existing file.java.io.IOExceptionprivate static void incBusy(java.lang.String path)
private static void decBusy(java.lang.String path)
private static int busyCount(java.lang.String path)
private static void openBackupDirectory(java.lang.String filename)
throws java.io.IOException
java.io.IOExceptionprivate static void cleanBackupDirectory()
throws java.io.IOException
java.io.IOExceptionpublic static int getPhase1Count()
public static int getPhase2Count()
static void incPhase1(boolean update)
static void decPhase1(boolean update)
static void incPhase2(boolean update)
static void decPhase2(boolean update)
static void updateBuildStatus()