Custom scheduling is the ticket to large-scale data distribution in groupware environments
David Yavin
On the small scale for which it was originally designed, Notes replication works like magic. For instance, if you work for a small company and you're on the road, you might dial in to your office's Notes server to synchronize copies of a Notes database. In a matter of minutes, every change to the database since your last replication appears in the database copy on your laptop. Except for the occasional busy signal, the process works like a charm.
Large Notes operations don't always feel so charmed, however. The problems all lie in scale, and a typical large Notes installation may have several thousand active Notes databases distributed among hundreds of Notes servers worldwide. By
design, any one of these databases can be updated by any of its users on any of the servers, from anywhere, and at any time.
In such a dynamic environment, keeping crucial information current is a daunting task. But it's not impossible. The secrets lie in proper scheduling and giving the responsibility of replication to the right people. It also means that there's no "one size fits all" solution. The right answers to the question of data replication in your organization are as unique as your fingerprints.
Small-Time Replication
When Lotus designed replication into Notes, it did so with the model of a small number of servers and a small geographical distribution in mind. Few people foresaw that within a decade, Notes replication worldwide would grow to the point where entire consulting companies would be dedicated to making it work better.
That's because many distributed organizations today choose to have local access to a
copy
of global information rather
than global access to centrally stored information. Regardless of the distributed-data setting--and there are several--replication is the underlying process by which multiple copies of the same data are synchronized, creating the illusion that distributed users are all sharing one set of data. Replication clearly serves as a means of overcoming technological and geographical boundaries among distributed members of an organization or workgroup.
Ken Lownie, president of Connexus Consulting Group (Andover, MA), helps his clients find ways to put Notes to work in their organizations. "By mobilizing the data, you can cut down on the mobilization of the experts," he says, referring to the reduced travel costs and more efficient use of personnel. "This not only cuts the cost of large roll-up processes, such as an audit, but it improves the quality of the information as well." On the less tangible side, "Notes can help a decentralized organization appear to clients as a single, well-coordinated company," he s
ays.
More a document repository than a database, Notes delights users with features not found in any other distributed-database product. But at the same time, Notes frustrates system managers and system administrators with unrivaled complexity and reliability problems. With Microsoft Exchange on the horizon, performance may become a key factor in the race to dominate the groupware market. But for now, the unique, document-based design of Notes propels it way ahead of the pack.
Another Notes advantage is its ability to give you a structured approach to unstructured data. Notes documents are flexible enough to be convenient to use, and they have just enough structure added to make them convenient to group, manage, sort, view, and distribute.
Mobile workers appreciate the support that Notes offers; the automation of a sales force is a popular example. The bidirectional exchange of information between a home office and a salesperson in the field can be reduced to pushing a button on the way o
ut to dinner. While the salesperson dines, his or her laptop remains in the hotel room, diligently replicating data with the home-office server. When dinner's finished, the laptop has pulled all the latest information on product status, pricing, and availability, while the server has received fresh information concerning new opportunities and transactions of the day's sales.
200 Points of Complexity
When you scale up the model to match a large Notes installation, the numbers become awe-inspiring. For instance, just draw 200 points on a piece of paper and consider how you might connect them. Then assign a time zone to each one and think about
when
you might connect them. Replications in a large organization are initiated according to a fixed schedule. The schedule determines the logical replication topology by stating which pairs of servers should replicate with each other. In this case, the schedule determines which of the two servers should initiate the replication an
d at what time. Both the complexity and importance of designing a reliable, "custom-fit" replication schedule are hard to overstate. Here are some of the main constraints that you'll face.
The duration of replications.
Notes replication takes a long time: minutes or hours, not seconds or milliseconds. The overhead of executing the process often creates more of a bottleneck than the volume of data does. Moreover, the duration of replications can differ at both ends. A hub's replicator can remain idle and not initiate a scheduled replication if the telephone line is tied up while the partner/server from the last replication continues to pull its updates.
The single-threaded replicator.
A server can pull updates from only one other server at a time. Whenever it is pulling, it cannot initiate additional replications and will not respond to replication requests sent by other servers.
Time-zone constraints.
For international organizations, scheduling across time zones
poses a major challenge. For example, the low-usage lunchtime hour might seem like a good time to schedule replications. However, the difference between 11:00 a.m. and noontime in New York is the difference between getting updates to European users before or after the end of their business day.
The Right Topology
In addition to devising the right replication schedules, there are two other factors to consider. First, ongoing monitoring and a periodic, systematic review of the entire replication system are of paramount importance. As Notes matures within an organization, us-age and needs grow dramatically. Such growth often renders the existing topology and replication schedule obsolete. This is especially true in the transition from the pilot phase to the substantial rollout phase. On average, the entire replication strategy should be reviewed annually.
Second, every Notes system has its own unique personality and needs a personalized replication strategy. Too often, the
numbers of servers and users in an organization serve as the main parameters that determine the organization's topology and replication-schedule requirements.
But each Notes implementation is unique in size, infrastructure, capabilities, and needs. Some companies use Notes to disseminate a small core of information that is generated in one central location. Others have massive, highly interactive applications in which documents are generated and modified by users scattered all around the globe. In some organizations, 24 to 48 hours for the propagation of updates is sufficient; others need several cycles per day. Some organizations commit substantial resources to monitoring performance, while others dedicate minimal resources.
Each replication strategy is unique as well. The objective is to strike a balance between simplicity and efficiency. Replication-schedule designs can provide efficient data propagation, but efficient schedules and the topologies that go with them might be complex in design
and might require extensive time and effort to manage.
There are two major points to keep in mind. First, while hub-and-spoke is the simplest topology, it's also the least efficient. (For a discussion of its inefficiencies, see "Optimizing Notes Replication," September 1994 BYTE.) Second, no matter how decentralized an organization may be, its replication strategy must be centrally planned, or at least centrally coordinated. In planning a schedule, the entire system should be examined, not just individual, isolated pockets. For example, one remote-site manager's decision to save a few hundred dollars by using a slow modem can tie up a central hub's line for hours every day and throw an entire organization's schedule off track.
Who's in Charge?
Neither large-scale system diagnostics nor the planning of enterprise-wide replications are tasks for Notes administrators. If Notes were the construction industry, administrators would be charged with making sure everything is ex
ecuted according to plan. Planning, inspecting, and ensuring performance would be the domain of architects and engineers. Too many organizations fail to realize this and thus settle for simple hub-and-spoke topologies and inefficient, often unreliable replication schedules.
On the small scale for which Notes was conceived, replication is a nonissue. The right replication schedule and administration can give large-scale operations a manageable feel once again.
WHERE TO FIND
Gupta Corp.
Menlo Park, CA
(800) 444-8782
(415) 321-9500
fax: (415) 321-5471
Lotus Development Corp.
Cambridge, MA
(800) 346-1305
(617) 577-8500
fax: (617) 253-9150
Microsoft
Redmond, WA
(800) 426-9400
(206) 882-8080
fax: (206) 936-7329
Sybase
Emeryville, CA
(510) 922-3500
fax: (510) 658-9441
Trinzic Corp.
Waltham, MA
(800) 234-7724
(617) 891-650
0
fax: (617) 622-1544
David Yavin holds a Ph.D. in mathematics from MIT in the fields of topology and combinatorics. He is president of DYS Analytics (Newton, MA) and works as a consultant for large Notes sites on issues of topology and replication scheduling. He can be contacted on the Internet at
david@math.mit.edu
or on BIX c/o "editors."
Replication's Fast Track
