Chapter 6

Replication of Directory information

6.1 INTRODUCTION

Replication of data is often beneficial to the users of distributed systems. The benefits come from: improved performance during retrieval, since copies of the data can be located nearer to the end users, and fewer users need to access each copy; and reliability, since if one copy is temporarily unavailable another one may be used in its place. However, as with all benefits in life, there is always an associated cost. You never get 'owt for nowt' as they say in Yorkshire. The cost comes from the added complexity of accessing the data, and from the overhead of keeping all the replicas up to date.

The global Directory service is no exception to this. There is added complexity in data retrieval, specifically in the management of knowledge references i.e. in knowing where all the copies of the data are located, and also in the need to avoid duplication in the result. The latter can occur in a Search operation that spans several DSAs, when each DSA has a copy of the required information, and each returns it. The precise way that this is avoided is described in § 9.13.14 and § 9.13.15. However, the most significant complexity is in updating the data. Three methods of updating may be used in distributed systems: update all the copies simultaneously, or have a voting scheme whereby only a majority of the copies need to be updated immediately, or designate one copy as the master and send all the updates to it. The first scheme is clearly unworkable in a globally distributed database, where systems are operating under different time zones, and it is impossible to guarantee that all systems will be operative all of the time. Voting schemes are suitable for LANs, but because of their added complexity in bringing all copies eventually up to date, are not really suitable for a global database. The master copy scheme was judged to be the best solution for the Directory, since concurrency control is greatly simplified. Furthermore, because the number of updates is anticipated to be small compared to the number of queries (this is certainly true for the telephone directory enquiries service), and updates are not deemed to be time critical, we can live with a master DSA i.e. the DSA holding the master copy of an entry, being occasionally and temporarily out of service.

The next question to ask is 'When will all the other (non-master) copies be updated?' Two solutions are possible: update them simultaneously with the master copy, or some time after it. The simultaneous scheme was not adopted, since this would then be the same as updating all the copies at once, and this was judged to be unacceptable for a globally distributed database. (Apart from the timing problems we would need a two phase commit protocol, and Directory experts did not want to employ this overhead.) Hence the Standard has adopted the scheme of updating the non-master, or shadow copies, sometime after the master. When this might be, is not specified in the Standard, and will be a time that is convenient to both the supplier and the receiver of the copy. Furthermore, the Standard does not insist that all the copies are updated at the same time. Some could be updated within milliseconds of the master being updated, others a few days later. The Standard refers to this scheme as providing weak consistency or transient inconsistency of the data.

So we now know that information may be copied between DSAs, that one copy is designated as the master copy, and that the remainder are called shadow copies. Data may be retrieved from any copy, but updates are always sent to the master copy, and are propagated some time later to the shadow copies. If a user suspects that a local shadow copy may not be up to date, then via the 'don't use copy' service control, they can request that the master copy is retrieved instead.

In order to cut down the workload of the master DSA, so that it does not have to update all of the shadow DSAs itself, a shadow DSA can also act as a supplier DSA to other shadow DSAs. This latter process of replication is termed secondary shadowing. If the master DSA provides the updates, it is termed primary shadowing. So not only can we have copies of the master copy, but we can also have copies of shadow copies. The shadowing mechanism is shown in Fig. 6.1. You can see that a tree of shadowing DSAs may be built up, with the master DSA as the root node. When a DSA is sending updates, it is called a shadow supplier DSA. When a DSA is receiving updates, it is called a shadow consumer DSA. As shown in Fig. 6.1, only the master DSA is never a shadow consumer.

As described in Chapter 4, there are two protocols that are used to support shadowing. The Directory Operational Binding Management Protocol (DOP) is used to establish a shadowing agreement (or relationship) between two DSAs, the shadow supplier and the shadow consumer. The Directory Information Shadowing Protocol (DISP) is used to transfer updates from the supplier to the consumer.

Fig. 6.1 Primary and Secondary Shadowing (A Shadowing Tree of DSAs).

6.2 SHADOWING AGREEMENTS

Before two DSAs can participate in shadowing, an operational agreement must be implicitly or explicitly reached between the administrators of the two DSAs. This agreement, which could be a legal contract between the two organisations, will cover all aspects about the shadowing such as: what data needs to be copied, what the data is to be used for, when the updates are to be provided, how much is to be charged for the service, what security policies must operate in the consumer DSA etc. The essential facts about the agreement then need to be fed into each of the DSAs. This can be done in one of two ways.

The DSA administrators could actually configure all of the necessary shadowing agreement information into their respective DSAs, via some administrator interface. After this has been done, a shadow Operational Binding will have been established between the two DSAs. In other words, the DSAs will now know about each other, and will also know what function each is expected to perform in order to correctly perform the shadowing.

Alternatively, the DOP may be used to establish an Operational Binding between the two DSAs. (An administrator might still need to alter the configuration of their DSA, so that it is prepared to accept a DOP Bind and an Establish Operational Binding request from the other DSA, but that apart, it is the Directory Operational Binding Management Protocol that formally establishes the shadow Operational Binding.) The Establish Operational Binding operation only carries the minimum information needed by the DSA software in order for it to participate in the shadowing. This minimum information is all that is required in order for a pair of DSAs to conform to shadowing as described in the Standard. This standardised information could well be only a subset of the operational details that were negotiated between the two organisations or DSA administrators. Other non-standardised information, such as accounting information, and penalties in case of failure to perform, are not included in this list. The standardised information is described later in § 6.3, but briefly it comprises of details of what information is to be shadowed, and when the updates are to be sent.

6.2.1 Inter-domain and Intra-domain Shadowing Agreements

The Standard recognises that two fundamentally different sorts of shadowing agreement are possible: inter-domain agreements and intra-domain agreements.

Inter-domain agreements would typically be negotiated on a one to one basis, between two different organisations (administrative domains), and would only be established when they are needed. The supplier might, for example, place tight restrictions on how the shadowed data is to be used. The agreement would probably be a legal document, and would contain more information than that carried in the Establish Operational Binding message. Money would almost certainly enter into the negotiations somewhere, and the consumer would have to pay for the privilege of receiving the copied information. The DSAs would probably need to be re-configured, in order to accept protocol messages from each other, because they would typically reject Establish Operational Binding or other messages from unknown DSAs. The DSAs may not even support the DOP, and might expect all the shadowing agreement information to be configured in prior to receiving the DISP messages.

Intra-domain agreements on the other hand, are set up between DSAs within the same domain. The parameters of the Establish Operational Binding message are usually, on their own, sufficient to allow the DSAs to participate in the shadowing. The domain could well be covered by a policy that allows DSAs within the domain to establish shadowing agreements between themselves with little or no administrative intervention. Legal agreements certainly will not be a pre-requisite to this sort of shadowing, and money probably will not be involved. For example, an organisation may have a policy that any DSA within its organisation may copy data from any other DSA within the organisation, subject only to the memory and disc capacity of the receiving DSA, and to its administrator kicking off an Establish Operational Binding message. In this case a supplying DSA does not need to refer to its administrator before replying in the affirmative to such a message. Simply logging that an agreement has been established may be enough. Shadowing is therefore reasonably well automated within the domain. The DOP is probably supported by most DSAs within the domain, and these may be pre-configured to expect Establish Operational Binding messages from any other DSA within the domain at any time. Intra-domain shadowing agreements can obviously be established between two DSAs without recourse to the DOP, but then some of the power is lost if administrators have to be involved in configuring up their DSAs before they can participate in shadowing.

6.3 PARAMETERS OF THE DOP ESTABLISH OPERATIONAL BINDING REQUEST

Either the shadow consumer or the shadow supplier may establish the shadow Operational Binding via the DOP. The first operation to flow between the pair of DSAs will be the DOP Bind operation. This establishes the connection between the DSAs, and allows each DSA to authenticate the other. Its parameters are identical to those of the Bind operation described in § 5.2. The only difference is that this operation will tell A-ASSOCIATE that it is the DOP protocol that is to be exchanged on the association, rather than the DAP, DSP or DISP.

Once the association is in place, the Establish Operational Binding request can be transferred. As described in Chapter 9, all Establish Operational Binding requests contain six arguments:

Table 6.1 The parameters of the Shadowing Agreement
Parameter Described in 
The data that is to be shadowed (unit of replication)  6.3.1
The mode of update  6.3.4 

The Establish Operational Binding reply consists of four arguments:

6.3.1 The Unit of Replication

Any part of the DIT may be replicated, ranging from a single attribute of a single entry, to a complete naming context. The 'unit of replication' is a syntactic construct that allows the part of the DIT that is to be replicated, to be precisely defined. The parameters that make up the unit of replication are listed in Table 6.2.

The context prefix unambiguously identifies the naming context that is to be fully or partially replicated.

Table 6.2 Parameters of the Unit of Replication
Parameter
The Context Prefix of the naming context 
The entries that are required from the naming context (this is called the replicated area) 
The attributes that are wanted from the replicated area 
The knowledge references that are wanted from the naming context 

The replicated area describes which entries from the naming context are to be shadowed. It does this using the subtree specification technique that we met in Chapter 2 (but see w/w 6.2). Only part of the naming context may be replicated, but the default is to replicate all the entries in the naming context (since this is the default for subtree specification). The subtree specification component base is a name, at or below the root entry (context prefix) of the naming context, at which the replication is to start. The components chop and specification filter may also be used to further select precisely those entries from the naming context that are required. Entries which have not been selected, but which have entries below them which have been selected, will be passed as glue DSEs to the consumer DSA. This is so that there are no unusual 'holes' in the replicated area. Fig. 6.2 shows a naming context that is to be partially replicated. The base is below the context prefix, and chop has been used to exclude the lower part of the naming context.

Fig. 6.2 Derivation of Shadowed Information by the master.

Once the entries have been selected, the attributes from the entries can be selected. The default is to select all the user attributes (including collective attributes) from all the entries. However, a subset of the attributes can be selected if desired. Attribute selection is based on the object classes of the entries in the replicated area. Attribute selection is constructed using a list of attribute selection statements. Each statement comprises an object class, along with the attribute types that should be included or excluded from copies of the entries that match this object class. Any attribute types can be specified alongside a particular object class. (They do not need to be part of the object class definition. This is because an entry can belong to several object classes and super-classes and any one of these may be used to select the entry.) If no object class accompanies a particular set of included and excluded attribute types, then these included or excluded attributes apply to all the entries in the replicated area. In this way it is possible to copy commonly used attributes, such as the telephone number and X.400 O/R address, from a whole set of entries of different object classes. Less useful attributes, such as room number and photograph, can be omitted from entries of specific or all object classes. Fig. 6.3 pictorially displays the options available in the construction of one of the attribute selection statements.

Fig. 6.3 Selection of attribute types to be shadowed.

If an attribute supertype is specified in an attribute selection statement, then all of its subtypes are automatically implied in the inclusion or exclusion. Attribute selection is cumulative, so that if an entry matches two of the object classes in the list of attribute selection statements, then the attributes that are copied will be derived from both statements. (In the case of a conflict specific 'includes' override specific 'excludes', which override implicit includes, but see w/w 6.3.) If a consumer has selected all the user attributes (including the collective attributes) from an entry, then the supplier will mark the entry as attribute complete. This will aid the consumer DSA in answering user interrogation requests (see § 5.4.1 for more details).

Finally, the knowledge references from the naming context can be selected (w/w 6.1). Either master or shadow or both types of knowledge reference can be copied. If a consumer has de-selected a portion of the naming context, via say a 'chop before' construct, never-the-less it can still request the subordinate (and non-specific subordinate) references below this, via an 'extended knowledge' boolean. (Though why it should quite like to do this has never been adequately explained to me!) Fig. 6.2 shows extended knowledge selection. The subordinate references have been tacked onto the bottom of the replicated area via glue DSEs.

For each entry (DSE) supplied to the consumer, the supplier may optionally mark it as subordinate complete. In order to qualify for this status, the shadowed entry (DSE) must be accompanied by all of its subordinates, as either entries or subordinate references. This flag aids the consumer during name resolution, and also during the evaluation of the List operation.

The shadow consumer is given, via the DISP, a copy of the whole of or just part of the naming context, as agreed. In either case, the part that it gets is called the shadowed information. The shadow consumer may subsequently act as a (secondary) shadow supplier, using its shadowed information as the data to be shadowed. However, the secondary consumer still specifies the unit of replication based on the original complete naming context. This unit of replication can only be the same as or smaller than the supplier's shadowed information, otherwise the secondary shadowing will have to be refused. You can see that it is possible for a naming context to be successively nibbled away as it passes through several secondary shadowing DSAs.

6.3.2 Shadowing Subentries and Operational Attributes

Relevant subentries are automatically provided by the supplier, without them being explicitly requested in the unit of replication. Generally speaking, relevant subentries are those subentries whose subtree specifications cover entries and attributes that are included in the replicated area. Note that the copying of a subentry also implies that the administrative entry must be partially copied as well i.e. the 'administrative role' (§ 2.11.2) and 'SubentryACI' (§ 8.3) operational attributes at least. Fig. 6.2 shows that the subentry governing the replicated area is included as part of the shadowed information.

Since shadow copies have to appear to the interrogation operations to be identical to the master copy, then access control information, collective attributes and subschema information usually must be copied.

Access control subentries holding PrescriptiveACI attributes (§ 8.3) which control access to entries and attributes within the unit of replication, must be transferred to the consumer. This is because a shadow DSA must be able to apply the same access controls as the master DSA. However, only the ACI which control interrogation of the Directory information need be transferred, since shadow DSAs are not allowed, per se, to update shadow entries irrespective of the ACI. Some security policies may not allow updateACI to be copied, and so the master DSA will have to filter these out from the subentry before sending it to the consumer. Note that if a proprietary access control scheme is in use, then shadowing can only take place between implementations that support the same access control scheme, or that can map between schemes.

Collective attribute subentries must be supplied if the unit of replication specifies all user attributes, or if the collective attribute types are explicitly included or are not implicitly excluded by the attribute selection statements.

The complete schema subentry will normally be copied to the consumer. Whilst the Standard only mandates that those subschema policy attributes which control the entries and attributes specified in the unit of replication need be copied, in most cases this will be some of the values of most of the policy attributes, and it will not be worth the processing time of de-selecting those that do not need to be copied.

Several Directory operational attributes are always copied to the consumer, irrespective of the actual user attributes selected from an entry. These are:

If any other Directory or DSA operational attributes are needed by the consumer, they must all be individually specified via 'include' attribute selection statements. The object class of an include statement may be absent (implying operational attributes from all entries) or present (implying operational attributes from entries of that particular object class).

6.3.3 Made to Measure or 'Off the Shelf'

You can see that the consumer can theoretically tailor the unit of replication to be just what it requires. Of course, whether or not a supplier is willing to provide made to measure replicas for every DSA that wants one, is another matter. There will obviously be a certain amount of processing and storage overhead by the supplier DSA, if it is to individually determine which data and which updates should be sent to which consumers. (There is also the non-negligible cost of implementing this flexible scheme!) Instead, a supplier DSA may operate much more like a supermarket, and have several off the shelf units of replication that it is willing to supply. These might typically be whole naming contexts, or very popular (commonly usable) subsets of them. A consumer DSA must then tailor these itself when it receives them. This may be done by the consumer DSA throwing away information that has been provided via the DISP, but for which it has no use. It will be interesting to see what the market decides in this respect.

6.3.4 The Mode of Update

This parameter of the shadowing agreement specifies when updates to the shadowed information should be sent from the supplier DSA to the consumer DSA. The parameter specifies who should initiate the updates, as well as the frequency at which the updates should be sent (Fig. 6.4). The initiator may be the supplier or the consumer, and the DISP protocol consequences of this are shown in Fig. 4.11.

Fig. 6.4 The Modes of Update.

Supplier initiated updates may be sent immediately that they have been received by the supplier (on change). For example, the master DSA receives a ModifyEntry request, and this causes it to initiate a Coordinate Shadow Update operation on its consumers. The precise meaning of 'immediate' is not standardised. The master could send a Coordinate Shadow Update to its consumers before replying to the original modify operation (e.g. ModifyEntry), or it could send it immediately after applying the modification. This is up to the DSA implementation. The Standard provides no mechanism for the consumer to rollback a shadow update, or for the user to abort a modification operation, and so users and implementors may assume that if the master DSA is able to successfully apply the modification, then nothing can stop it from eventually completing throughout the entire tree of shadows. If a shadow is temporarily unavailable, the supplier should retry the Coordinate Shadow Update operation after a suitable time period. This time period will be for the DSA implementor to choose.

Fig. 6.5 Periodic generation of update windows.

Periodic updates can be initiated by either of the DSAs. Periodic updates define a repeating window in time, during which updates will normally be transferred. The windows are defined using three parameters: a begin time (absolute), a window size (in seconds) and an update interval (also in seconds). The way that these three parameters interact to define the windows is shown in Fig. 6.5.

If othertimes has been selected, this allows the initiating DSA to request/send updates outside of the periodic windows. Othertimes can be used for example, by a supplier that wishes to intermittently send updates.

Finally, it should be noted that the selection of a mode of update is not legally binding. It is only a statement about the preferred mode of update. There is nothing to stop either of the DSAs initiating updates at other times, as the operational need arises. Similarly, there is nothing to stop either DSA from refusing to participate in a shadow update operation, whether or not it conforms to the preferred mode of update. There are thus no conformance requirements to the Mode of Update.

6.3.5 Role Specific Parameters

Role specific parameters contain information which one DSA needs to pass to the other DSA. They are always sent by the same DSA, irrespective of which DSA in the pair initiates the DOP request (§ 9.16.4). This contrasts with the parameters of the shadowing agreement, which are always sent by the DSA initiating the DOP operation.

The shadow consumer has no role specific parameters that it needs to pass to the shadow supplier.

The shadow supplier has two role specific parameters that it needs to pass to the shadow consumer. If the shadow supplier is the initiator, then the two parameters are included in the Establish Operational Binding request. If the shadow supplier is the responder, then the two parameters are included in the Establish Operational Binding response.
Note. The Standard has no role specific parameters defined. These two parameters have been erroneously classified as part of the shadowing agreement. A defect report on this was submitted to the Standards committee, but rejected because it would destabilise the protocol.

The first of these parameters is the Access Point of the master DSA. The supplier may optionally tell the consumer which DSA holds the (master) naming context that is being copied. This will aid the consumer in two ways. Firstly, if the consumer receives an update request for the shadowed information that it holds, it may immediately forward this to the master, thereby speeding up the process of name resolution. Secondly, if an interrogation request arrives, in which the user has set the 'don't use copy' service control, it may likewise send it straight to the master DSA.

The second parameter is a boolean flag which tells the consumer whether or not the supplier would like to be told about any secondary shadows that the consumer directly or indirectly supplies. Support for this service by the consumer is optional. It does not have to forward the information to the supplier, but if the flag is set to False, then it must not forward the information to the supplier (see also w/w 6.6).

6.3.6 Parameters of the Establish Operational Binding Response

These are described fully in § 9.16, but essentially they consist of: confirmation of the binding type and binding ID, the Access Point of the recipient (this is to be used by the initiator on all subsequent Bind requests to the recipient), and the recipient's role specific parameters (listed in § 6.3.5). No further information is sent in the response.

6.4 UPDATING A SHADOWING AGREEMENT

Clearly administrators may re-negotiate a shadowing agreement at any time. The Standard does not place many restrictions on which parameters of the agreement can be re-negotiated. Clearly the Standard cannot restrict changes to those aspects of the agreement that are not standardised, such as tariffs and uses of the copied data. However, the Standard does place one restriction on the changes that can be made to the parameters of the shadowing agreement carried in the DOP.

Whilst most parameters of the agreement can be changed, the one exception to this is that the base entry of the replicated area cannot be changed - this would need to have a new agreement established and the existing one terminated.

Given this limitation, it is possible to change the agreement in order to: increase or decrease the size of the replicated area, shadow a fewer or a greater number of attributes, change which knowledge references are wanted, and alter the mode and frequency of sending the updates.

If the DOP is not being used, administrators will need to re-configure their DSAs to reflect the new arrangements. If the DOP is used to maintain the shadowing agreement, a Modify Operational Binding request, containing the revised parameters, is sent from one of the DSAs to the other one (the direction is immaterial). The revised parameters are either accepted or rejected by the other DSA. If the parameters are rejected, a preferred set of parameters may optionally be returned in the Operational Binding Error message (§ 9.16), but it is not the intention of the Standard that changes to a shadowing agreement should be negotiated via the DOP. If negotiation is required, this should preferably take place off-line by the administrators. The Modify Operational Binding request is intended to notify the other DSA of already agreed or innocuous changes to the shadowing agreement.

Obviously care needs to be exercised in requesting changes to an agreement, particularly if the consumer is also supplying secondary shadows. These latter agreements could be invalidated by an inappropriate change to the consumer's agreement. For example, if a shadow consumer requests less attributes from now on, it is no longer able to supply these to any of its secondary shadows.

6.4.1 The Modify Operational Binding Operation

The Modify Operational Binding operation is primarily used to update the shadowing agreement. However, it may also be used to update the Access Points of the two DSAs involved in the shadowing, as well as the Access Point of the master DSA that is held by the consumer, and the Access Points of the secondary shadows that are held by the supplier.

As described in Chapter 9, the Modify Operational Binding operation request consists of seven parameters:

Either the shadow consumer or the shadow supplier may send a Modify Operational Binding request to the other DSA. The Modify Operational Binding response is either an OK message with no parameters, saying that the modification has been accepted, or is an error message with a parameter, saying that the modification has been refused for the reason given (§ 9.16).

6.4.2 Role Specific Parameters

The only parameter that the shadow supplier is specifically responsible for updating, is the Access Point of the master DSA. This was optionally passed to the shadow consumer when the shadow Operational Binding was first established. If the master DSA is re-configured, then its new Access Point will need to be passed to all the consumer DSAs that hold a copy of it. The Modify Operational Binding operation passes this information from a supplier DSA to a consumer DSA, as the shadow supplier role specific parameter. Successive invokes of the Modify Operational Binding operation allow the new address of the master DSA to be cascaded down through the tree of shadow DSAs, from the master DSA to all of the secondary shadow DSAs. Each consumer, that is also a supplier of secondary shadows, will invoke a Modify Operational Binding operation on its consumers, after receiving the operation from its supplier.

A similar, though reversed, process is also used for updating the list of secondary shadows held by a supplier DSA. A shadow consumer can tell its supplier the names and addresses (Access Points) of all the secondary shadows that it is supplying. Secondary Shadows is the consumer's role specific parameter of the Modify Operational Binding operation. As new secondary shadowing agreements are established by a consumer (this time acting in the role of supplier), it can tell its supplier about them. The supplier then adds its own secondary shadows to this list, and ripples this information back to its supplier (who adds its own secondary shadows), so that eventually the master DSA may obtain a complete list of all of the secondary shadows that are receiving copies of its naming context (Fig. 6.6). This process is repeated every time shadowing agreements are created and terminated. Note that only the Access Points of DSAs holding 'commonly usable' shadows are notified to a supplier, and then only if the supplier indicated that it wanted them when the agreement was first established.

Obviously the roles of the two DSAs cannot be switched midway through the shadowing agreement, i.e. the supplier cannot suddenly become the consumer and vice versa (but see w/w 6.7).

Fig. 6.6 Updating the master with secondary shadows.

6.5 TERMINATING A SHADOWING AGREEMENT

Termination of a shadowing agreement can take place at any time, and can be initiated by either party. When the agreement was initially forged, the two DSA administrators would have determined what action each should take upon termination. For example, the consumer DSA may be allowed to hold the shadowed information as a cache, that will slowly become out of date, or it may be required to delete the shadowed information immediately. However, all of these details are outside the scope of the Standard.

If the DOP is not being used to manage the Operational Binding between the two DSAs, then termination will have to be effected by re-configuration of the DSAs. If the DOP is being used, then the Terminate Operational Binding request is sent from one DSA to the other (the order is immaterial). This operation consists of four parameters as described in § 9.18:

Termination should be accepted by the recipient DSA. There is no defined error message (such as 'sorry, please give me another chance') that allows the recipient DSA to refuse to accept a properly constructed termination message.

Once termination has been affected, the consumer DSA must terminate any secondary shadowing agreements still in force that depend upon this now defunct shadowed information. Clearly the consumer is no longer able to continue to update its consumers. In this way termination can ripple down the tree of shadowing DSAs.

6.6 UPDATING THE SHADOWED INFORMATION

Shadowed information is updated by the Directory Information Shadowing Protocol (DISP). As mentioned in Chapter 4, this protocol comprises of DSA Shadow Bind and Unbind operations, plus three other operations: Coordinate Shadow Update and Request Shadow Update (either of which serve to prepare both DSAs for the transfer of the shadowed information), and Update Shadow (which transfers the shadowed information from the supplier to the consumer) - see Fig. 4.11. Either of the first two messages serve to synchronise both DSAs.

6.6.1 Synchronising the DSAs

Both the Coordinate Shadow Update and Request Shadow Update messages carry the timestamp of the last batch of updates successfully sent by the supplier and/or received by the consumer. This allows the message recipient to see the state of the other party, and to respond accordingly. If their states are the same, an OK reply is sent, which will be followed by an Update Shadow message. If their states are different, an error reply is sent, with a diagnostic which gives the state of the recipient. This allows the initiator to send another synchronisation message with a revised timestamp, or optionally to request a total refresh if the consumer is completely out of synchronisation (w/w 6.4).

Both of the synchronisation messages contain four parameters, as described in Table 6.3.

Table 6.3 The parameters of the Coordinate Shadow Update and Request Shadow Update operations
Parameter Description 
Shadowing Agreement ID The unique identifier of the shadowing agreement (Operational Binding) 
Time of last batch of updates The timestamp allocated by the supplier to the last batch of updates that were successfully transferred 
Update Strategy  The type of update that should be sent. This is a choice between: 

       - no changes - (only allowed on Coordinate message)
       - incremental refresh - delta changes only
       - total refresh - all the shadowed information should be sent again, or
        - a proprietary mechanism

 See § 4.9 for a fuller description 

Security Parameters (optional) Only used if the operation is digitally signed (§ 5.4.2) 

Once the DSAs have been synchronised, updates to the shadow information can be transferred.

6.6.2 Transferring the Shadowed Information

The shadowed information is transferred from the supplier to the consumer via the Update Shadow operation. The parameters of this operation are shown in Table 6.4.

As was shown in Fig. 6.2, the shadowed information is derived from the supplier's DSA Information Tree, and is always a tree starting at the root of the DIT. This shadowed information tree is a tree representation of the information that is being shadowed, and may be dynamically generated by the supplier DSA prior to transfer. It will usually be a subset of the supplier's DSA information tree. The DSEs that comprise the shadowed information tree are actually termed shadowed DSEs, or SDSEs for short. The shadowed information tree is initially transferred in its entirety to the consumer DSA, immediately after the shadowing agreement is established i.e. a total refresh takes place. The tree is transferred SDSE by SDSE, starting at the root. Upon arrival at the consumer DSA, the SDSEs are merged into the consumer's DSA Information Tree, turning back into DSEs with the shadow DSE type bit set (§ 2.13.2, Table 2.1). SDSEs are never actually visible in either of the DSA's Information Trees.

Table 6.4 The parameters of the Update Shadow operation

Parameter Description 
Agreement ID The unique identifier of the shadowing agreement (Operational Binding) 
Update time  The timestamp allocated by the supplier to this batch of updates 
Next Update Window (optional) The window expected to be used by the supplier for the next batch of updates. (Only present if Update Mode of 'other times' was specified) 
Refresh Information The updates to the shadowed information. This takes one of four formats: 

            - no changes 
            - total refresh 
            - incremental refresh, or 
            - a proprietary mechanism 

Security Parameters (optional) Only used if the operation is digitally signed (§ 5.4.2) 

Which SDSEs need to be transferred in the protocol depends upon the update strategy. Obviously, if no changes are transferred, then no SDSEs will accompany the Update Shadow message. In this case the purpose of the message is only to update the batch timestamp. For example, periodic updates may have been agreed, but no data may have actually been updated in a particular period. Never-the-less, the supplier may send a 'no changes' update at the agreed time, so that the consumer knows that it has not missed an update.

If total updates are sent, then each SDSE from the root of the shadowed information tree will be transferred to the consumer DSA. Each SDSE will contain the complete set of attributes that are being shadowed. Obviously the very first time that shadowed information is sent to a consumer DSA, it will be a total update package. Subsequent updates may be either 'total' or 'incremental' updates, according to need. When a subsequent update is a total update, the consumer DSA must first dispose of all those DSEs in its DSA Information Tree that are present as a result of the current shadowing agreement. The SDSEs from the total update package may then be inserted into its DSA Information Tree.

If incremental updates are transferred, then only those SDSEs that have been updated since the last Update Shadow message was sent, plus their parents up to the root, need to be included. Parent SDSEs that have not been updated since the last message, are only included in the current Update Shadow message so that the updated SDSEs can be linked into the root of the DIT. Only the relative distinguished names of the parent SDSEs are included. Their attributes do not need to be sent. SDSEs that have been updated do not necessarily need to be sent in their entirety. Only the changes that have been made to them need to be transferred in the protocol.

Incremental updates are the most complicated to compute and to transfer. The supplier DSA needs to keep a record of every SDSE in the shadowed information tree that has been changed since the last Update Shadow operation was successfully sent. Each SDSE change that is carried in the protocol must be idempotent, so that if needs be, the update can be re-transmitted and applied more than once by the consumer, without affecting the integrity of its database i.e. its DSA Information Tree. This is needed in order to aid recovery.
Note. Idempotent (Latin for 'the same power') updates always have the same result, no matter how many times they are applied. An example of an idempotent change is 'set value to x'. A non idempotent change would be 'increment value by x'.
Thus SDSEs that have been deleted from the supplier's shadowed information tree still have to be transferred by name, but with a 'deleted' bit set (Fig. 6.7). SDSEs that have been added are transferred in their entirety. SDSEs that have been updated need to list the attribute types and values that have been added and removed. SDSEs that have been renamed need to be transferred by their old name, and quote their new name. Similarly SDSEs that have been moved need to be transferred by their old name and quote their new position in the DIT. (Their subordinates are not transferred if they have not been updated. The consumer will automatically move them as well.)

Incremental updates significantly reduce the volume of shadowed information that needs to be transferred, and if the number of updates is small, they can significantly reduce the amount of processing in both DSAs. However, you can see why some supplier DSAs may only be willing to supply whole naming contexts initially rather than 'made to measure' ones to each consumer DSA. It is much easier to subsequently determine which of the user's modification operations need to be transferred to which consumer DSA in a particular incremental update, since they will all receive all of them.

Fig. 6.7 Generating the Update Shadow Request.

Figure 6.7 shows the shadowed information tree which was generated when a shadowing agreement was first established. This would have been transferred in its entirety to the consumer DSA. It also shows the shadowed information tree that exists now, after two modifications have taken place. An entry has been deleted, and another one has had its attributes updated (both modified entries are shown with a tick). If a total refresh is to take place, then this new tree will be transferred in its entirety to the consumer DSA. If an incremental update is to take place, then an incremental tree logically needs to be built that shows the difference between the old and new shadowed information trees. The incremental tree depicts the SDSE changes that will be transferred in the protocol to the consumer DSA. Both the deleted and updated SDSEs are transferred, as well as the RDNs of all of their parent SDSEs. It is the responsibility of the consumer DSA to correctly reconstruct its DSA Information Tree based on the new shadowed information, regardless of whether total or incremental changes are transferred.

6.6.3 Using the Shadowed Information

Merging SDSEs into a consumer's DSA Information Tree is not always straightforward. A consumer DSA might (foolishly?) replicate different parts of the same naming context from two different supplier DSAs, and end up with overlapping replicated areas (w/w 6.5). However, even if this is not the case, there will be many instances when the top layers of the shadowed information tree overlap with the top layers of the consumer's own DSA Information Tree. This will often occur with administrative entries and subentries, for example, when a subordinate DSA replicates the naming context held in the immediately superior DSA (or vice versa). In this case, the context prefix and the subentries will be provided by both the HOB and the DISP. The Standard requires that in the case of overlap, the shadowed context prefix and subentries should logically be kept separate from the non-shadowed information, and should always be linked directly to the replicated area to which they appertain. This is so that the shadowed and non-shadowed information may both remain logically consistent. A simple rule of thumb is: do not mix and match shadowed entries with non-shadowed context prefixes and subentries.

Apart from this, shadowed information - both entries and knowledge references - can normally be used to satisfy user queries in just the same way as mastered information, since both are accurate snapshots of the DIT. The obvious exceptions to the use of shadowed information are: that shadowed information cannot be the subject of the modification operations, and the user may prohibit (via the 'don't use copy' service control) shadowed information from being used to satisfy their interrogation operations. Both shadowed and mastered entries can be used to satisfy the multiple entry interrogation operations List and Search, providing of course, that the above rule of thumb is not violated.

WEIRD AND WONDERFUL

6.1. The draft Standard allowed both immediate superior and subordinate knowledge references to be copied from a naming context. Due to last minute objections by some experts, immediate superior references were removed, so that now only subordinate knowledge references may be copied. This seems illogical.

6.2. Unfortunately 'subtree specification' is not based on naming contexts, but is based on subtrees of the DIT. Whilst DIT subtrees can be bigger than naming contexts, obviously the actual subtree that is copied from a DSA may not be bigger than the naming context from which it is derived. Thus an additional restriction has to be introduced into the meaning of a subtree specification, when it is used to describe a replicated area, and that is, that it cannot exceed the naming context. Thus if the subtree specification merely contains a base entry, although this implies all the entries below base down to the leaves, it is effectively bounded by the circumference of the naming context, i.e. if the naming context does not extend down to the leaves, then only those entries that are in the naming context will be replicated. Thus it is not possible to imply replication from two of more DSAs (or naming contexts) via one unit of replication. Two separate shadowing agreements would need to be established, one for each DSA (or naming context), with each having its own unit of replication.

6.3. Interestingly enough, the Standard says that a supplier DSA may, in exceptional circumstances, omit to transfer to the consumer DSA certain attributes that are included in the unit of replication, if security policies dictate this. So in principle it would be possible to have a conformant supplier DSA that always supplies nothing to the consumer DSA, irrespective of the unit of replication!

6.4. The Standard does not define any procedures for recovery in the case of a failure during the updating of a shadow. However, all the protocol messages are already defined in order to aid re-synchronisation. All that is needed is some text saying which messages should be sent under which circumstances, and what actions should be taken when errors are noted. This is clearly an omission from the Standard, but it will have to be left either to the functional standards groups to define their own recovery procedures, or to the defect mechanism to update the Standard, or to implementors to define their own procedures (clearly the least favourable solution).

6.5. If a consumer DSA replicates different parts of the same naming context from two different supplier DSAs, it could end up with overlapping replicated areas. It would be provided with two sets of SDSEs, some of the SDSEs could be common to both sets but might have different contents. The Standard says that in this case the consumer DSA must hold the two replicated areas in separate information planes, and must not merge them together. It seems that the full implications of this scenario have not been properly thought through.

6.6. An early draft version of the replication protocol had an agreement parameter which forbade a shadow from supplying the shadowed information to secondary shadows. This parameter was subsequently removed from the protocol, as it was argued that this was part of the contractual arrangements between the two parties, and was not essential to the operation of the DISP between the supplier and the consumer.

6.7. Early work on the '93 Standard proposed facilities to allow the master DSA to be dynamically changed into one of the primary shadowing DSAs (and vice versa). Part of the mechanism that would have been used to achieve this would have been a Modify Operational Binding with a role reversal parameter.