your multitiered architecture. This article presents some tricks and tips that will help you get started with automated testing.
Your testing should provide you with detailed -- and calming -- information on reliability (Will your system lock up with increased load?), scalability (Will access time degrade with increased load?), and design flaws (Can the system architecture handle the level and complexity of use?). This last point is important: the problem with many client/server systems is, indeed, their architecture. Load testing helps isolate architectural problems. To achieve the desired results, you are, of course, better off optimizing your system's architecture up front and not as an afterthought. To optimize your design, you should do the following.
-- Minimize the amount of network traffic.
One way to do this is to encapsulate the SQL or use remote procedure calls (RPCs) instead of send
ing raw SQL across the network.
-- Insulate the whole system from errant or failed processes.
One failed client should not lock up others.
-- Reduce the contention for resources.
For example, in a database client/server system, reduce the length of transactions to avoid database-resource contentions.
Getting Testy
By this point, you should have the bug population mostly quelled and be ready to take on the more systemwide and performance-associated aspects of the application. To create an
automated test
in a client/server environment, you need both a test harness (which provides the infrastructure in which the tests run) and one or more test scripts.
A test harness schedules a test to run at a specific time, for a specific time interval, on a specific machine on a network. A commercially available test harness typically also includes an integrated test-development environment. A test harness collects information
at run time, including details on failures, performance statistics, and network status.
In general, there are four ways to create a test. You can record the actions of real users on your GUI front end and then replay those actions later. Similarly, you might capture the post-GUI application output directly and play that back later. (For example, to test a Web server, you might capture the HTTP requests from the client to the server and then replay them.) You can programmatically generate one or more test scripts (e.g., scripts that capture the low-level graphical subsystem protocol instead of recording user actions that screen resolutions and fonts can affect). Finally, you can write a
white-box
test (explained below) that talks to your system's API directly and mimics user actions.
Thus, tests that you create and run in the test harness typically perform several tasks. They might drive a GUI front end to simulate the actions of a live user. They might also interact directly
with a server by circumventing the GUI front end: This is called a white-box test. In addition, they might execute the protocol between the client and the server (e.g., SQL and HTTP requests).
Most tests must fit your specific application; therefore, you cannot buy shrink-wrapped tests from a vendor. But there are tools available that simplify the process of creating your own tests.
Automated client/server harness-and-test solutions fall into four categories, depending on whether each part is shrink-wrapped or custom-developed. Custom test-and-harness combinations are the most flexible, but naturally they require the most work on your part. If the combination seems suited to your particular applications, shrink-wrapped tests and harnesses minimize your effort. Otherwise, a custom test with a shrink-wrapped harness provides flexibility and control without much effort on your part. Pairing a shrink-wrapped test with a custom harness is the least desirable option.
To determine the type of solut
ion that you should use to load-test your application, you should consider several factors. First, determine the expected number of users. Second, consider the type of information clients and servers will exchange. It might be simple, static information, like HTTP; complex information, like SQL; or some higher-level protocol, like RPC. Finally, determine the number and complexity of tiers in a multitiered architecture.
Shrink-Wrapped Tests and Harnesses
Our experience indicates that it's more cost- and time-effective to use a shrink-wrapped harness. If you're testing a multitiered product where you need to simulate many users, we recommend using a custom white-box test. A two-tiered system with a small number of users using a standard protocol, such as HTTP or SQL, tests best with a shrink-wrapped test harness and test-development environment.
New technologies are emerging from vendors of automated test tools. For example, in the high-end client/server tools market, Pure Softwar
e's PurePerformix Script-and-Go technology allows you to capture the interaction between your application and the input device, such as a character terminal, an X Window System terminal, or a PC. This is a variant of record/playback technology, which captures the low-level subsystem protocol. PurePerformix can also capture SQL traffic between your client and database server. The captured SQL traffic goes into scripts later. PurePerformix can arrange a mix of scripts to run from one central driver machine.
Client/server applications can manage a complex information life cycle, where data is entered, processed, and delivered throughout the corporate enterprise. Softbridge's Automated Test Facility (ATF) addresses the demands of the multitiered, multiapplication architecture. You can run ATF from a single control machine to follow the life cycle of a data record. If the process fails, ATF stops the failed test and starts the next test. ATF supports both record/playback and programmatic test development.
Mercury's LoadRunner works in one of two modes. In GUI Virtual User mode, LoadRunner works with Mercury's automated test tool, WinRunner, to provide a way for you to run WinRunner tests in a distributed fashion on multiple PC clients through the client GUI front end. Mercury supports both recording and playback of user actions, of low-level protocols, and of programmatically developed test-script technology.
In LoadRunner's DB Virtual User mode, the tests run white-box tests or prerecorded script files (e.g., SQL or HTTP). You can generate queries and updates to your server over the network without going through your client GUI front end. The advantage of white-box testing is that you can run multiple virtual users on each machine. (Mercury claims you can have multiple GUI virtual users, but you might run into problems with memory constraints and setup issues.)
On networked Unix machines, LoadRunner can generate multiple virtual database users that send SQL (or other types of traffic) directly
to the server. Using multiple virtual database users you can simulate hundreds or (in theory) thousands of users with the right hardware. Mercury also provides tools to collect SQL from your client for subsequent use in DB Virtual User tests. This technology can do either white-box testing or recording and playback of SQL.
Segue Software's QualityWorks products also simulate multiple virtual users. QA Partner:Distributed allows you to run scripts on multiple client GUI front ends simultaneously (with the added benefit that clients can be on different platforms). QA DBTester can send SQL directly to a database server, thereby circumventing your client front end.
SQA's LoadTest
works in conjunction with the company's Robot automated test tool. You drive the PC client front end with Robot scripts that LoadTest distributes to machines over the network. The Robot scripts load your server with queries and updates. Your limit is the number of PCs that are available or that you're w
illing to set up and use for testing. (With this system, the number of PCs limits the number of virtual users.) LoadTest therefore supports the recording and playback of user interaction in a distributed situation.
Using PurePerformix, QA DBTester, or LoadRunner's DB Virtual User technology, you can simulate large numbers of users to verify the scalability of your client/server architecture. You can also do this with Mercury Interactive's GUI Virtual User technology or SQA's LoadTest technology (using one virtual user per PC in the distributed test).
Both distributed GUI testing and distributed multiprocess virtual-user testing capture detailed information for later analysis. In some cases, you can write the script with functions that provide timing information. Of course, your test harness tells you when a specific test fails.
The disadvantage of simulating multiple users on a single machine is that you're not generating real network-traffic conditions. But it's easier to simulate a large n
umber of users. You can generate real network-traffic conditions by running one virtual user per PC through the client front end. But this process is too labor-intensive and expensive -- you need more than 1000 machines.
There are other pitfalls when simulating multiple users on a single machine. Primarily, you're running a small number of PCs and predicting the performance degradation from the results of the simulation. If the degradation is predictable, you're fine. But if performance degrades, your test done with 20 machines will not reveal the serious degradation that occurs with 21 machines.
We recommend using multiprocessed virtual users and GUI virtual users to verify scalability. The only alternative is to work with an outside testing lab to perform massive single-user-per-PC tests; this is an expensive proposition, but it might be the only way in certain cases.
Custom Tests and Harnesses
A number of situations mandate a custom rather than a shrink-wrapped solution
. Where flexibility and control are paramount, you'll want to at least consider writing the tests yourself. Such situations include an application that's out of the mainstream, platform combinations that exceed the abilities of the shrink-wrapped tools, and performance characteristics that are special cases. The downside is that it requires more effort to develop and verify these tests.
You can write custom test solutions completely in-house, and tools are available to help you do just that. For example, Database Accessory (DA), from Integrated Computer Solutions, takes you through the whole client/server applications development process. The tests deliver keystrokes to the GUI using a scripting language called Tool Command Language (TCL, pronounced "tickle"). These scripts drive the application and hence simulate user input. TCL can query the database independently and match the internal state of the application with the results of the independent database query.
This verification method offers e
rror checking that might be appropriate in any number of custom automated test scenarios. You can use DA's method with most of the shrink-wrapped automated test solutions, but typically you get all your feedback through the client GUI front end.
Blood, Sweat, and Tiers
Another reason to go the custom-test route is if the communication mechanism between your client and server uses technology that shrink-wrapped tools cannot effectively capture. Then, writing white-box tests may be the only way to go (other than using GUI-automated scripts). Our company, Quintus, uses an RPC protocol between each tier in the company's multitiered architecture. But shrink-wrapped tools cannot capture RPCs, so we decided to write our own white-box tests instead.
One advantage of white-box testing in a multitiered architecture is that you can use the APIs between each tier to test their interaction. To build white-box tests for Quintus's customer-information management applications suite, we took adv
antage of internal APIs while using LoadRunnner's harness capabilities.
Quintus tested the interaction not only between the white-box client and the intermediate servers but also between those servers and the database. This ensured that each tier was tested independently from the other tiers and was healthy. The white-box tests also simulated the same network traffic between the servers and the real clients.
The listing
"A Sample Test"
shows a code snippet of a white-box test written against Quintus's APIs. This snippet shows the use of the LoadRunner functions that captured transaction information and sent messages to a central controller. The white-box tests were designed to perform typical operations that a user might perform on the keyboard: creating a new record, updating an existing record, and deleting a record.
Quintus also wrote a number of tests that simulated the client/server interaction process, such as connecting and disconnecting from the database and ini
tializing the client subsystem. Simple scripts repeated the tests a specified number of times. Other tests exercised each API call independently, allowing testers to gauge their behavior under stress.
When configuring database servers and networks, it was necessary to ensure that the database servers could handle several hundred connections. This involved configuring the database server machine with enough memory, storage, and shared memory.
Don't Be a Crash-Test Dummy
The use of automated test tools can save time and effort, but only if you let them. When using automated test tools in the software development process, take the path of least resistance. For example, don't waste resources trying to make your test tool "see" custom controls in the target application's GUI. If control type mappings (e.g., mapping a custom button to the type button) don't fix the problem, you can create automated test scripts that don't use the errant control.
Another way to maximize your engi
neering resources is to start small in client/server automated testing. Start a pilot program with modest expectations and plans. As you garner some success in your automated testing, widen the scope and depth of the testing.
When running your tests, you need information about what the application is doing while the tests are running. Consider how best to monitor the progress of your automated tests in real time, using information such as database and intermediate server log files; user connections; record, page, and table locks; and log files generated by the actual tests.
Ideally, you should install a network sniffer to monitor the flow of packets on the network to identify any bottlenecks. Don Lanoue, a senior technology consultant at Digital Network Services, says that while a half-second response is fine, if you're a support agent on the phone, you need instant answers. Your network bandwidth can make the difference.
Lanoue, who is a veteran of modeling small and large systems, explains
that "there are customers out there that have networks that don't run well. We go in and collect data and feed it into a modeling tool. Once you have the network modeled, then you can play 'what if.'"
But Lanoue warns that not just any system administrator can effectively accomplish this task. If you're considering using a modeling tool, check out CACI's ComNet III and Systems and Networks' Bones. Both tools do LAN and WAN network simulations.
Put to the Test
If you plan to deliver client/server software, an automated test harness and test-development tool should be a part of your test process. Our experiences at Quintus lead us to strongly recommend spending your resources on test development and not on test-harness development.
Quality and scalability in a client/server system must be built into the architecture. Make sure you understand the intricacies of your architecture and allocate the time and resources needed to perform automated testing on it. Write white-box te
sts that simulate multiple users from a single machine. We leveraged our testing effort by using tests simulating multiple users. Finally, if you're using automated tests in a client/server environment, always start small. Do a pilot of your test, and when that works, scale up. This method works.
Where to Find
Integrated Computer Solutions
Cambridge, MA
Phone: (617) 621-0060
Internet:
http://www.ics.com/IntraNet
Testing, Testing
