public final class SyncRunner
extends java.lang.Object
implements java.lang.Runnable
This class is used in the Ganymede server to handle Sync
Channel-style incremental and full-state synchronization. Unlike
the full-state GanymedeBuilderTask system, SyncRunner
is not designed to be customized through subclassing. Instead, all
SyncRunner objects generate a common XML file format for writing
out synchronization data. SyncRunner objects are registered in the
server for execution automatically when a Sync Channel
object (using the syncChannelCustom DBEditObject subclass
for management) is created in the Ganymede data store. This
Sync Channel object provides all of the configuration
controls which determine where the XML synchronization files are
written, what external program should be used to process the files,
and what data needs to be written out for synchronization.
Every time a transaction is committed, the Ganymede server
compares all objects involved in the transaction against every
registered Sync Channel object. If any of the objects
or fields created, deleted, or edited during the transaction
matches against a Sync Channel's so-called Field
Options matrix of objects and fields to sync, that Sync
Channel will be tasked to write an XML file to the directory
configured in the Sync Channel's Queue Directory
field.
For incremental Sync Channels, each XML file that is created in a Sync Channel output directory is given a unique filename, based on the number of transactions committed by the Ganymede server since the server's database was first initialized. The first transaction committed is transaction 0, the second is 1, and so forth. These transaction numbers are global in the Ganymede server. Every time any transaction is committed, the transaction number is incremented, whether or not any Sync Channels match against the transaction.
These incremental XML files are written out synchronously with the transaction commit. What this means is that the transaction is not counted as successfully committed until all SyncRunners that match against the transaction successfully write and flush their XML files to the proper output directory. If a problem prevents any of the SyncRunners from successfully writing its XML files, the transaction will be aborted as if it never happened. All of this is done with proper ACID transactional guarantees. The SyncRunner implementation is designed so that the Ganymede server can be killed at any time without leaving a transaction partially committed between the Sync Channels and the Ganymede journal file. Either a transaction is successfully recorded to all relevant Sync Channels and the journal, or it will not be recorded to any of them.
Whenever any transaction is successfully committed, each
incremental Sync Channel Runner is scheduled for execution by the
GanymedeScheduler. When the Sync
Runner's run() method
is called, it runs its external Service Program,
passing the most recently committed transaction number as a single
command line argument. The Service Program is meant
to examine the Queue Directory specified in the
Sync Channel object, and to process any XML files with
numbers less than or equal to its command line argument. As is the
case with GanymedeBuilderTask, the GanymedeScheduler will not
relaunch a SyncRunner until the previous execution of the
SyncRunner completes. Unlike GanymedeBuilderTask, the Ganymede
server does not prevent new files from being written out while the
Service Program is being executed. It is the
responsibility of the Service Program to ignore any
XML files that it finds with transaction numbers higher than that
passed to it on the command line.
In this way, the incremental Sync Channel system allows
transactions to be committed at a rapid rate, while allowing the
Service Program to take as little or as much time as
is required to process transactions. The principle of back-to-back
builds is very much part of this Ganymede synchronization mechanism
as well.
Because the incremental Sync Channel transaction files are
generated while the transaction is being committed, the Sync
Channel writing code has complete access to the before and after
state of all objects in the transaction. This before and after
information is incorporated into each XML file, so that the
external Service Program has access to the change
context in order to apply the appropriate changes against the
directory service target.
Full-state automatic sync channels, by contrast, work more like the GanymedeBuilderTask system. When a transaction is committed that matches a full-state sync channel's Sync Channel object definition, the sync channel is scheduled for later execution in the Ganymede scheduler. When the full-state sync channel is serviced by the scheduler, the full state of the system (as filtered by the requirements of the Sync Channel object at the time the scheduler runs) the XML file is generated and the service program is run immediately therafter.
Incremental sync channels can be more efficient, as only changed data need be written to the Sync Channel, but not all types of data services can be supported with incremental synchronization.
Because incremental synchronization is based on applying changes to a directory service target, it works best on directory services that can be updated incrementally, like LDAP. It is not designed for systems that require full-state replacement in order to make changes at all, such as NIS or classical DNS. Even for systems that can accept incremental changes, however, the use of discrete deltas for Ganymede synchronization can be problematic. If an XML transaction file cannot successfully be applied to a directory service target, the Ganymede server has no way of knowing this, because the Service Program is not run until sometime after the transaction has been successfully committed.
In order to cope with this, the Sync Channel system has provisions for doing manual 'full-state' XML dumps as well. You can do this manually by using the Ganymede xmlclient. The command you want looks like
xmlclient -dumpdata sync=Users > full_users_sync.xml
The effect of this command is to dump all data in the server that matches the filters for the 'Users' Sync Channel to the full_users_sync.xml file. Note that in order for this to work, you should make sure that your Sync Channel's name does not include any whitespace. Java's command line parsing logic is fundamentally broken, and makes it impossible to portably process command line parameters with internal whitespace. Note as well that, as with all xmlclient dump operations, the database is locked against transaction commits while this is running, so this operation can only be done by a supergash-level account.
You would use such a full state sync file for those cases where the incremental synchronization system experiences a loss of synchronization between the Ganymede server and the target directory service. The idea is that your external Service Program should in some way be able to recognize that it is being given a full state dump, and undertake a more lengthy and computationally expensive process of reconciliation to bring the directory service target into compliance with the data in the Ganymede server.
See the Ganymede synchronization guide for more details on all of this.
| Modifier and Type | Class and Description |
|---|---|
static class |
SyncRunner.SyncType
Enum of possible modalities for a SyncRunner
|
| Modifier and Type | Field and Description |
|---|---|
private booleanSemaphore |
active
This semaphore controls whether or not this SyncRunner will
attempt to write out transactions.
|
(package private) static boolean |
debug |
private java.lang.String |
directory |
private QueueDirFilter |
dirFilter
A filter we use to find transaction files in the queue directory.
|
private java.lang.String |
fullStateFile |
private scheduleHandle |
handle
A reference to the scheduleHandle for this sync channel.
|
private boolean |
includePlaintextPasswords |
(package private) static byte |
major_xml_sync_version
XML version major id
|
private SyncMaster |
master
If the Sync Channel object in the Ganymede server had a class
name defined, and we can load it successfully, master will
contain a reference to the
SyncMaster used to provide
augmentation to the delta sync records produced by this
SyncRunner. |
private java.util.Map<java.lang.String,SyncPrefEnum> |
matrix |
(package private) static byte |
minor_xml_sync_version
XML version minor id
|
private SyncRunner.SyncType |
mode
Controls what type of Sync Channel we're handling.
|
private java.lang.String |
name |
private booleanSemaphore |
needBuild
This variable is true if we've seen a transaction that
requires us to issue a build on the next run().
|
private java.lang.Object |
queueGrowthMonitor
The lock object we're using for the queueGrowthSize management.
|
private int |
queueGrowthSize
If we're an incremental sync channel, we'll track the size of
what we add to the queue during runs of the service program, so
we can determine whether the service program cleared the queue
(minus the new entries) during its run.
|
private java.lang.String |
serviceProgram |
private int |
transactionNumber |
(package private) static TranslationService |
ts
TranslationService object for handling string localization in
the Ganymede server.
|
| Constructor and Description |
|---|
SyncRunner(DBObject syncChannel) |
| Modifier and Type | Method and Description |
|---|---|
void |
checkBuildNeeded(DBJournalTransaction transRecord,
DBEditObject[] objectList,
DBEditSet transaction)
This method checks this Full State SyncRunner against the
objects involved in the provided transaction.
|
private XMLDumpContext |
createXMLSync(DBJournalTransaction transRecord)
This private helper method creates the
XMLDumpContext that writeIncrementalSync() will
write to. |
java.lang.String |
getDirectory()
Returns the queue directory we'll write to if we're an
incremental build channel.
|
java.lang.String |
getFullStateFile()
Returns the file we'll write to for a full state build.
|
java.lang.String |
getName()
Returns the name of the Sync Channel that this SyncRunner was
configured from.
|
private SyncPrefEnum |
getOption(DBField field)
Returns the SyncPrefEnum, if any, for the given
field.
|
private SyncPrefEnum |
getOption(short baseID)
Returns the SyncPrefEnum, if any, for the given base.
|
private SyncPrefEnum |
getOption(short baseID,
short fieldID)
Returns the SyncPrefEnum, if any, for the given base and field ids.
|
int |
getQueueSize()
Performs a readdir loop on the queue directory for this sync
channel (if incremental) to see how many entries are currently in
the queue.
|
java.lang.String |
getServiceProgram()
Returns the name of the external service program for this Sync
Channel.
|
int |
getTransactionNumber()
Returns the number of the last transaction known to have been
safely persisted to disk (journal and sync channels) at the time this
method is called.
|
boolean |
includePlaintextPasswords()
Returns true if the Sync Channel attached to this SyncRunner
is configured to write plain text passwords.
|
private void |
initializeFieldBook(DBEditObject[] objectList,
FieldBook book)
This method creates an initial internal FieldBook for this
SyncRunner, based on the parameters defined in the incremental
Sync Channel DBObject that this SyncRunner is configured
from.
|
boolean |
isFullState()
Returns true if this SyncRunner is configured as a full state
sync channel.
|
boolean |
isIncremental()
Returns true if this SyncRunner is configured as an
incremental sync channel.
|
boolean |
mayInclude(DBField field,
boolean hasChanged)
Returns true if the given type of field may be included in
this sync channel.
|
boolean |
mayInclude(DBObject object)
Returns true if this sync channel is configured to ever
include objects of the given baseID.
|
boolean |
mayInclude(short baseID)
Returns true if this sync channel is configured to ever
include objects of the given baseID.
|
boolean |
mayInclude(short baseID,
short fieldID,
boolean hasChanged)
Returns true if the given type of field may be included in
this sync channel.
|
void |
run()
This is the method that will run when the GanymedeScheduler
schedules us for execution after a transaction commit.
|
void |
runFullState()
This method handles running a full state XML build.
|
private void |
runFullStateService()
This method executes the external build, feeding the external
service script the name of the full state XML file that we dumped
out.
|
void |
runIncremental()
This method handles running an incremental XML build.
|
void |
setActive(boolean state)
This method is used to enable or disable this sync channel's
writing of transactions to disk.
|
void |
setScheduleHandle(scheduleHandle handle)
Sets a reference to the scheduleHandle that this sync channel
will use to communicate its status to the admin consoles.
|
void |
setTransactionNumber(int trans)
Used to set the number of the last transaction known to have been
safely persisted to disk (journal and sync channels) at the time this
method is called.
|
boolean |
shouldInclude(DBEditObject object)
Returns true if the DBEditObject passed in needs to be synced
to this channel.
|
boolean |
shouldInclude(DBField newField,
DBField origField,
FieldBook book)
Returns true if the given field needs to be sent to this sync
channel.
|
java.lang.String |
toString() |
void |
unSync(DBJournalTransaction transRecord)
If we go to commit a transaction and we find that we can't
write a sync to its sync channel for some reason, we'll need to
go back and erase the sync files that we did write out.
|
private void |
updateAdminConsole(boolean justRanQueue)
Updates the queue status in the admin consoles
|
private void |
updateInfo(DBObject syncChannel)
Configure this SyncRunner from the corresponding Ganymede Sync
Channel DBObject.
|
private void |
writeFullStateSync(GanymedeSession session)
This method writes out a full state XML dump to the fullStateFile
registered in this SyncRunner.
|
void |
writeIncrementalSync(DBJournalTransaction transRecord,
DBEditObject[] objectList,
DBEditSet transaction)
This method writes out the differential transaction record to
the delta Sync Channel defined by this SyncRunner object.
|
static final TranslationService ts
TranslationService object for handling string localization in the Ganymede server.
static final boolean debug
static final byte major_xml_sync_version
static final byte minor_xml_sync_version
private java.lang.String name
private java.lang.String directory
private java.lang.String fullStateFile
private java.lang.String serviceProgram
private int transactionNumber
private boolean includePlaintextPasswords
private java.util.Map<java.lang.String,SyncPrefEnum> matrix
private SyncRunner.SyncType mode
Controls what type of Sync Channel we're handling.
private booleanSemaphore needBuild
This variable is true if we've seen a transaction that requires us to issue a build on the next run().
We set this to true on startup so that we will initiate a "catch-up" build on server startup, just in case.
private booleanSemaphore active
This semaphore controls whether or not this SyncRunner will attempt to write out transactions. This semaphore is disabled when this Sync Channel is disabled in the admin console.
private SyncMaster master
If the Sync Channel object in the Ganymede server had a class
name defined, and we can load it successfully, master will
contain a reference to the SyncMaster used to provide
augmentation to the delta sync records produced by this
SyncRunner.
private scheduleHandle handle
A reference to the scheduleHandle for this sync channel. We use this handle to set the status for propagation to the admin console(s).
private int queueGrowthSize
If we're an incremental sync channel, we'll track the size of what we add to the queue during runs of the service program, so we can determine whether the service program cleared the queue (minus the new entries) during its run.
private java.lang.Object queueGrowthMonitor
The lock object we're using for the queueGrowthSize management.
private QueueDirFilter dirFilter
A filter we use to find transaction files in the queue directory.
public SyncRunner(DBObject syncChannel)
private void updateInfo(DBObject syncChannel)
Configure this SyncRunner from the corresponding Ganymede Sync Channel DBObject.
public void setTransactionNumber(int trans)
Used to set the number of the last transaction known to have been safely persisted to disk (journal and sync channels) at the time this method is called.
public int getTransactionNumber()
Returns the number of the last transaction known to have been safely persisted to disk (journal and sync channels) at the time this method is called.
public void setActive(boolean state)
This method is used to enable or disable this sync channel's writing of transactions to disk.
public java.lang.String getName()
Returns the name of the Sync Channel that this SyncRunner was configured from.
public boolean isFullState()
Returns true if this SyncRunner is configured as a full state sync channel.
public boolean isIncremental()
Returns true if this SyncRunner is configured as an incremental sync channel.
public java.lang.String getDirectory()
Returns the queue directory we'll write to if we're an incremental build channel.
public java.lang.String getFullStateFile()
Returns the file we'll write to for a full state build.
public java.lang.String getServiceProgram()
Returns the name of the external service program for this Sync Channel.
public void setScheduleHandle(scheduleHandle handle)
Sets a reference to the scheduleHandle that this sync channel will use to communicate its status to the admin consoles.
public void writeIncrementalSync(DBJournalTransaction transRecord, DBEditObject[] objectList, DBEditSet transaction) throws java.io.IOException
This method writes out the differential transaction record to the delta Sync Channel defined by this SyncRunner object. The transaction record will only include those objects and fields that are specified in the Sync Channel database object that this SyncRunner was initialized with, or which are included by a SyncMaster class referenced by name in the SyncChannel DBObject used to create this SyncRunners.
transRecord - A transaction description record describing
the transaction we are writingobjectList - An array of DBEditObjects that the transaction
has checked out at commit timetransaction - The DBEditSet that is being committed.java.io.IOExceptionpublic void checkBuildNeeded(DBJournalTransaction transRecord, DBEditObject[] objectList, DBEditSet transaction) throws java.io.IOException
This method checks this Full State SyncRunner against the objects involved in the provided transaction. If this SyncRunner's Sync Channel definition matches against the transaction, a flag will be set causing a Full State build to be executed upon the next run of this Sync Runner in the Ganymede scheduler.
transRecord - A transaction description record describing
the transaction we are checkingobjectList - An array of DBEditObjects that the transaction
has checked out at commit timetransaction - The DBEditSet that is being committed.java.io.IOExceptionpublic void unSync(DBJournalTransaction transRecord) throws java.io.IOException
If we go to commit a transaction and we find that we can't write a sync to its sync channel for some reason, we'll need to go back and erase the sync files that we did write out. This method is responsible for wielding the axe.
transRecord - A transaction description record describing
the transaction we are clearing from this sync channeljava.io.IOExceptionprivate XMLDumpContext createXMLSync(DBJournalTransaction transRecord) throws java.io.IOException
This private helper method creates the XMLDumpContext that writeIncrementalSync() will
write to.
transRecord - A transaction description record describing
the transaction we are writingjava.io.IOExceptionprivate void initializeFieldBook(DBEditObject[] objectList, FieldBook book)
This method creates an initial internal FieldBook for this SyncRunner, based on the parameters defined in the incremental Sync Channel DBObject that this SyncRunner is configured from.
public boolean includePlaintextPasswords()
Returns true if the Sync Channel attached to this SyncRunner is configured to write plain text passwords.
public boolean mayInclude(short baseID)
Returns true if this sync channel is configured to ever include objects of the given baseID.
This method does not take into account any augmentation done
by a linked SyncMaster.
public boolean mayInclude(DBObject object)
Returns true if this sync channel is configured to ever include objects of the given baseID.
This method does not take into account any augmentation done
by a linked SyncMaster.
public boolean shouldInclude(DBEditObject object)
Returns true if the DBEditObject passed in needs to be synced to this channel.
Only intended to be used for Full State Dumps, as it doesn't include the SyncMaster augmentation for incrementals.
public boolean shouldInclude(DBField newField, DBField origField, FieldBook book)
Returns true if the given field needs to be sent to this sync channel. This method is responsible for doing the determination only if both field and origField are not null and isDefined().
Only intended to be used for Full State Dumps, as it doesn't include the SyncMaster augmentation for incrementals.
public boolean mayInclude(DBField field, boolean hasChanged)
Returns true if the given type of field may be included in this sync channel. The hasChanged parameter should be set to true if the field being tested was changed in the current transaction, or false if it remains unchanged.
This differs from shouldInclude on DBField in that this method leaves it to the caller to decide whether the field has changed.
public boolean mayInclude(short baseID,
short fieldID,
boolean hasChanged)
Returns true if the given type of field may be included in this sync channel. The hasChanged parameter should be set to true if the field being tested was changed in the current transaction, or false if it remains unchanged.
This differs from shouldInclude on DBField in that this method leaves it to the caller to decide whether the field has changed.
private SyncPrefEnum getOption(DBField field)
Returns the SyncPrefEnum, if any, for the given field. This enum can be one of three values:
NEVER, meaning the field is never included in this sync channel, nor should it be examined to make a decision about whether a given object is written to this sync channel.
WHENCHANGED, meaning that the field is included in this sync channel if it has changed, and that the object that includes the field should be written to the sync channel if this field was changed in the object. If a field has an option string of WHENCHANGED but has not changed in a given transaction, that field won't trigger the object to be written to the sync channel.
ALWAYS, meaning that the field is always included in this sync channel if the object that it is contained in is sent to this sync channel, even if it wasn't changed in the transaction. If this field was changed in a given transaction, that will suffice to cause an object that is changed in any fashion during a transaction to be sent to this sync channel. In this sense, it is like WHENCHANGED, but with the added feature that it will "ride along" with its object to the sync channel, even if it wasn't changed during the transaction.
If WHENCHANGED or ALWAYS is true for a field in a given object type, the corresponding SyncPrefEnum for the object's type should be WHENCHANGED, signaling that at least some fields in the object should be sent to this sync channel when the object is involved in a transaction.
private SyncPrefEnum getOption(short baseID, short fieldID)
Returns the SyncPrefEnum, if any, for the given base and field ids. This enum can be one of three values:
NEVER, meaning the field is never included in this sync channel, nor should it be examined to make a decision about whether a given object is written to this sync channel.
WHENCHANGED, meaning that the field is included in this sync channel if it has changed, and that the object that includes the field should be written to the sync channel if this field was changed in the object. If a field has an option string of WHENCHANGED but has not changed in a given transaction, that field won't trigger the object to be written to the sync channel.
ALWAYS, meaning that the field is always included in this sync channel if the object that it is contained in is sent to this sync channel, even if it wasn't changed in the transaction. If this field was changed in a given transaction, that will suffice to cause an object that is changed in any fashion during a transaction to be sent to this sync channel. In this sense, it is like WHENCHANGED, but with the added feature that it will "ride along" with its object to the sync channel, even if it wasn't changed during the transaction.
If WHENCHANGED or ALWAYS is true for a field in a given object type, the corresponding SyncPrefEnum for the object's type should be WHENCHANGED, signaling that at least some fields in the object should be sent to this sync channel when the object is involved in a transaction.
private SyncPrefEnum getOption(short baseID)
Returns the SyncPrefEnum, if any, for the given base. This option string should be one of two values:
NEVER, meaning that this object type will never be sent to this sync channel.
WHENCHANGED, meaning that this object type may be sent to this sync channel if any field contained within it which has a SyncPrefEnum of WHENCHANGED or ALWAYS was changed during the transaction.
public void run()
This is the method that will run when the GanymedeScheduler
schedules us for execution after a transaction commit. If our
incremental flag is set to true, we'll consider ourselves as
servicing an incremental build, in which case we just call our
servicer script with the last transaction committed as our
command line argument. If we are full state, we'll do more like
a GanymedeBuilderTask| and
write out a complete, filtered dump of the server's contents to
an XML file and then call the servicer with that file name as a
command line argument.
run in interface java.lang.Runnablepublic void runFullState()
This method handles running a full state XML build.
private void writeFullStateSync(GanymedeSession session) throws NotLoggedInException, java.rmi.RemoteException, java.io.IOException
This method writes out a full state XML dump to the fullStateFile
registered in this SyncRunner. It is run within the context of a
DBDumpLock asserted on the
Ganymede DBStore.
NotLoggedInExceptionjava.rmi.RemoteExceptionjava.io.IOExceptionprivate void runFullStateService()
This method executes the external build, feeding the external service script the name of the full state XML file that we dumped out.
public void runIncremental()
This method handles running an incremental XML build.
public int getQueueSize()
Performs a readdir loop on the queue directory for this sync channel (if incremental) to see how many entries are currently in the queue.
private void updateAdminConsole(boolean justRanQueue)
Updates the queue status in the admin consoles
public java.lang.String toString()
toString in class java.lang.Object