Picker International takes advantage of expert-system technology to wring maximum leverage from its field-service operation
Scott Wallace
It wasn't all that long ago that field engineers from Picker International headed out each day with a pager, some test equipment, and a car crammed full of three-ring binders thick with schematic diagrams, parts catalogs, and other product documentation. The Cleveland-based billion-dollar-a-year provider of medical diagnostic systems was respected for the quality of its field service, but the prospects for continued improvement and opportunities to leverage the company's experience and growing knowledge base were limited.
With service revenues accounting for one-fifth of its annual income, and with health-care product purchases slowed by cost-containment pres
sures and regulatory uncertainty, the last thing Picker wanted was to restrict its service opportunities. The company concluded that more effective capture, management, and use of product and service information was critical to sustaining and expanding its service market. Central to the solution it devised is an expert system that captures key information from the company's phalanx of field engineers and makes this information available throughout the organization.
The Way We Were
A number of years ago, Picker licensed field-service dispatching software from Astea International (Bedford, MA) and customized it to support the company's field-service requirements. The software, Fieldwatch, ran on an IBM 3090 mainframe at Picker's centralized customer-support facility. Fieldwatch managed the formal dispatching operation and also handled associated functions (e.g., call accounting, inventory management, and billing).
Fieldwatch dispatched field engineers using a commercial paging service. The
engineers telephoned when they were paged. They were assigned a service call and provided with a description of the problem as reported by the customer. The field engineer called the customer and then went to repair the equipment. After the equipment had been worked on, the engineer telephoned the dispatch center and described all activities and parts associated with the repair. This information was entered into customer and service databases.
Information Repositories
These databases are critical to Picker's ability to assess its field-service performance. They also provide important information concerning product reliability. ``We've accumulated a massive amount of information,'' said Nancy Booth, manager of service operations at Picker. ``We measure equipment performance, mean-time-to-repair, how quickly we respond to customers, and we track hardware failures. This helps us identify components that need to be fixed or improved and allows us to design products that are not only more reliable bu
t more readily repaired.''
The databases contained valuable, comprehensive information about the customer site and all past maintenance and repair activities. But this was available only at the central facility and was not accessible to an engineer in the field. If field-service personnel could access historical service documents, they would be better prepared to resolve problems. And if they could dial into the customer's Picker equipment (many Picker products support dial-in capability, and future systems are slated to incorporate this support), they could test and diagnose the system remotely--even before visiting the site. The company decided to invest in laptops for each field engineer. It also obtained software to support remote access to databases and expanded dispatching services.
All this pointed to the need to restructure dispatching to provide access to a range of product, customer, and service documents and records. ``About a year ago,'' said Booth, ``we implemented nationwide a PC r
emote interface to our dispatching system and service databases. Now, when a field engineer gets a page, he dials into our mainframe computer from his laptop PC and receives all open service calls.'' At the same time, past service histories for each call can be downloaded to the laptop.
The new laptop-enabled dispatching accomplished more than simply improving the amount and availability of service and repair information. It significantly reduced personnel requirements at the central dispatching facility. ``The field engineer is no longer calling the customer report center to log activities,'' said Booth. That resulted not only in far fewer operators and data-entry personnel but in fewer data-entry errors.
There was an additional dividend. ``Our Fieldwatch system is on an IBM 3090 mainframe--an environment where purchase, maintenance, and development costs run very high. With PCs, you can do a lot more for a lot less money,'' Booth said. Picker spent nearly $2.5 million dollars enhancing its dis
patching system and buying more than 900 Toshiba T4500s to run a combination of off-the-shelf and custom-developed applications and utilities (see the figure ``Field-Service Laptop'' on page 88).
The company is saving a million dollars a year in personnel and other dispatching-center costs as a result of that investment. It intends to leverage the service and repair information collected to both reduce equipment operating costs and improve equipment and service performance.
Expert-System Quest
With field-service laptops deployed and dispatching enhanced, Picker had in place the technological foundation needed to provide field engineers with Questor, its expert-system diagnostic support. Built on TestBench, the Carnegie Group's (Pittsburgh, PA) expert-system software, and populated with Picker's knowledge base, Questor guides field engineers through the diagnosis and repair of Picker products. Because Questor provides links to on-line documentation and because it presents a diagnostic appr
oach developed and refined by Picker subject-matter experts, the system makes diagnostic procedures in the field more uniform and more successful.
Questor is a Windows-based application that presents the field engineer with a decision-tree architecture for problem analysis and repair. Each limb on the tree includes a ``why'' window that describes the purpose of the proposed procedure and a ``how'' window that explains exactly how to execute the procedure. Questor also includes a notepad, where field engineers can record observations and capture errors in or improvements to Questor.
The contents of the notepad, as well as a log of the diagnostic pathway traversed during the repair, are stored on the laptop. They can be transmitted to centralized service databases for review and, if appropriate, incorporation into the next release of Questor. This feedback loop between Questor users and designers not only provides for improvements in the product, it permits the collection of observations concernin
g diagnostic and repair processes and procedures. When aggregated and analyzed, these observations can be used to drive process improvements throughout the field-service organization.
Knowledge-Base Architecture
Picker relied on a variety of resources when developing the on-line knowledge bases for each of its supported products. Service-engineering specialists, offering both engineering and manufacturing expertise, as well as regional and district specialists, collaborated with knowledge engineers to articulate diagnostic and repair strategies. However, developing tactics and tools to support those strategies has taken a while.
Picker first examined expert systems to support diagnostics and repair almost five years ago. ``We were in analysis paralysis for a good long time,'' noted Michael Grybush, manager of advanced services technology. Although it was stymied on exactly how to proceed, the company recognized the need for something like flowcharts and decided to start there. ``Copying t
he diagnostic-tree concept, a master product-diagnosis flowchart of 75 pages of `IFTHENELSE' with targets at the bottom of each page saying `Go to page XX' was built. It's a big, ugly, formal document that, if you had the patience, would take you close to the source of a problem,'' said Grybush.
Originally conceived of as a repository for practical diagnostic approaches, the flowchart served a broader purpose as well. ``It was a reasonable jumping-off spot to begin putting down the [knowledge-base] architecture because there's a direct correspondence between the flowchart and the structures used to build Questor,'' Grybush said.
The flowchart helped Picker knowledge-base developers ensure that the approach was not only diagnostically appropriate but was also comprehensive and presented a complete model. ``The whole thing is a huge state machine, and how you move from one state to the next depends upon where you are in the path and what your various inputs are,'' said Grybush.
The flowchar
t was critical in the early stages of the Questor knowledge-base development effort. As time went on, however, it became less representative of an optimal diagnostic pathway. This was partly because the chart was generated early in the development life cycle and partly because, as a paper-based document, it remained static while Picker's understanding of how its systems worked--and failed--grew over time.
Knowledge-Base Development
Picker knowledge engineers familiar with its products and skilled in the construction of knowledge bases used technical-service notes and on-line documentation to supplement the flowcharts and interviews with domain experts. ``The same knowledge-extraction process that one would use to create the troubleshooting flowchart is used to create the on-line diagnostic adviser,'' noted Grybush.
The development of Picker's knowledge base for its CT (computed tomography) product line took, for example, three months. During this time, one experienced knowledge engineer w
orked 300 hours researching and preparing the CT knowledge base in TestBench.
It could have taken a lot longer. ``TestBench's object-oriented approach greatly simplifies and speeds development,'' Grybush said, contrasting this with more error-prone and time-consuming programming methods. ``As you develop your knowledge base, moving objects around the decision tree is as simple as clicking and dragging, and the objects' attributes follow from one place to another.'' A Unix-based Sun SparcStation with 24 MB of memory, about 700 MB of hard drive space, a tape drive, and a large-screen color display supported the knowledge-base development. (The Carnegie Group will soon release a Windows-based knowledge-engineering environment.)
In theory, a knowledge engineer with no subject-matter expertise can extract knowledge from a domain expert, but Picker's experience recommended a different approach. ``We've found it to be much more advantageous if the knowledge engineers know what they are talking about,''
Grybush said. ``It simplifies interpretation and diminishes the misunderstandings that stem from using English.'' For technical expertise in the CT product line, the knowledge engineer drew upon one primary-domain expert (120 hours) and four secondary-domain experts (15 hours each). The resulting knowledge base includes 1400 objects and is encoded in a 1-MB binary file.
Objectifying Knowledge
Objects in the knowledge base include, symptoms, tests, repairs, decision points, questions, components, rules, and so on. ``An object might be an observed failure, such as `console will not boot,''' noted Grybush. Associated with objects are attributes. ``There might be a pointer to a text string or to a bit-mapped image. There might be another pointer to some text called `why,' to describe why you're being asked to do a procedure. These pointers are different, depending upon the type of object you're dealing with, and, in the strictest sense, the inheritance properties of objects follow smoothly through
this general object-oriented model.''
Some of the most useful objects in the knowledge base are electronic documents (see the text box ``Supporting Questor with Electronic Documents'' on page 94). These include hypertext product documentation, repair procedures, parts lists, block and schematic diagrams, and so on. ``Vectors are generated inside the Questor environment, and these point to a path, file, chapter, and page within a hypertext document. Once you're there, you can wander around the document in a hyperlinked fashion,'' Grybush said.
TestBench
TestBench not only supports Picker's electronic documents, it provides a prebuilt structure for diagnostic knowledge-base development. Rather than providing developers a generic expert-system shell, TestBench offers an object-oriented, diagnostic-specific development system that includes proven problem-solving strategies and a diagnostic methodology.
``It may be an empty shell in terms of the exact way to diagnose a particular proble
m, but the framework for diagnosing problems is built into the product,'' said Kenneth Kleinberg, research director for Applied Intelligent Systems at the Gartner Group (Stamford, CT). ``You have problems; you have ways of testing for them. You have ways of dealing with multiple paths and ways of running different tests and verifying that the repairs you made are accurate and complete.'' Grybush and other Picker employees credit this prefocus on diagnostic support with significantly simplifying the development and implementation of its expert system.
TestBench employs a series of problem-solving approaches appropriate to diagnostic situations. These approaches include decision-tree reasoning for structured but simple problems; fault-hierarchy reasoning for highly structured and complex problems; case-based reasoning for shallow, simple lookup types of problems; and rule-based reasoning for exception-oriented problems.
TestBench does not include a model-based-reasoning paradigm. ``This approach t
ends to be valuable when you understand a good deal about how a product functions, but not much about how it fails,'' said Kleinberg. ``A good example is circuit-board diagnosis, where the designed function of the component is well known, but the ways it could fail are many and unanticipated.'' Given that TestBench focuses on failures and their causes, model-based reasoning tends to have limited applicability for most prospective users.
TestBench Architecture
Structurally, TestBench has three components (see the figure ``TestBench Components'' on page 96): TestBuilder (composed of a Knowledge Editor and a Diagnostic Problem Solver), TestBridge (which translates the knowledge base developed with TestBuilder into files that can be accessed by TestView), and TestView (a run-time diagnostic procedure).
TestBench also includes a set of utilities for nondiagnostic-activity support. A log function captures the diagnostic procedure in ASCII format and generates statistical reports. A transcript f
unction records the diagnostic session screen-by-screen for postdiagnosis review. A notepad gives users a means of recording comments and observations for feedback to knowledge-base developers. A recording feature enables users to interrupt and resume a diagnostic session.
Although most users interact with TestView, the bulk of the effort to create an expert diagnostic system like Questor is spent developing and debugging the knowledge base. This object-oriented knowledge base is organized in a causal hierarchy. Observable symptoms reside at the top, and possible causes, either failures or ``cases,'' extend downward from the symptoms.
Each symptom is associated with its set of potential causes by a caused-by link. The knowledge base is organized as a network of failures or cases with failures having primary links to other causally related failures. Both objects (i.e., failures and cases with failures) have secondary links to other objects (e.g., tests and repairs), and the ordering of these link
s structures the diagnostic process.
TestBuilder
TestBuilder's Knowledge Editor builds and maintains the knowledge base, and its Diagnostic Problem Solver, or DPS, runs and debugs the knowledge base. Three levels of information can be distinguished, and the Knowledge Editor reflects and supports each. At the primary level is the outline of the hierarchical knowledge base. Clustered around each primary object in the hierarchy is secondary knowledge, consisting of tests, repairs, and rules that are associated with a selected failure or case. At the final level can be found additional descriptive and control information about the objects.
The Knowledge Editor supports two levels of expertise. In system-directed mode, the Knowledge Editor guides the developer in creating and defining objects in the knowledge base. In user-directed mode, the developer edits objects without intervention. The Knowledge Editor provides support for both graphic and textual editing (see the figure ``Knowledge Edito
r Interface'' on page 90). In graphic editing, an ``object'' cursor is used to position each object.
Case objects are linked to symptoms or grouped under case-clusters. Failures can be graphically linked to other failures, indicating what the cause of the problem is. When the knowledge base is edited graphically, the textual view is updated; when the knowledge base is edited textually, the graphic view is likewise updated.
The DPS uses knowledge databases and technician input to search out the cause of the problem being diagnosed. The cause may be determined by successful search of the failure hierarchy, or the technician may determine the cause using case-based reasoning. Problem text matching (which is driven by a natural-language interface or by a menu search of the knowledge base) enables the DPS to bypass the problem classification process and focus directly on the problem under investigation.
The DPS allows problems to be classified using fault-hierarchy reasoning or case-based reas
oning. Fault-hierarchy reasoning evaluates candidate failures by assigning a confirmed, disconfirmed, or unknown state to each fault, proceeding through confirmed faults until a cause with no caused-by links is found. Using case-based reasoning, cases are tested and assigned scores that provide a level of probability or belief for each case.
The case score is determined by the developer of the tests. Because cases cannot have ``children,'' they are considered to be the repairable object in the hierarchy. When a failure cause is confirmed, TestBench suggests the appropriate repair and helps the field engineer validate that the repair has been successful.
TestBridge and TestView
For a knowledge base to be accessed, it must be converted for use on the appropriate platform. This procedure has three steps. First, knowledge-base files are compiled by TestBuilder into a single file (GKB) that is system-independent and usable by multiple platforms. Next, TestBridge transfers the GKB file from the
development environment to the delivery system. Finally, the GKB file is converted from generic format to a binary file readable by the delivery system. The knowledge base is then ready for processing by TestView.
TestView is the run-time component of TestBench; it supports the same inferencing process performed by the DPS, but for the end user rather than the developer. TestView is organized in a modular structure that the Carnegie Group calls its Kernel Application Architecture. At the core is the TestView kernel; layered upon that are the Application Interface Module and the User Interface Module.
This layering offers developers maximum flexibility, enabling them to customize applications readily, to replace the User Interface Module with their own interface, and to embed diagnostic knowledge-base applications in their own software environments. The current version of TestView runs with a C kernel and has Visual Basic application- and user-interface layers.
The goal of TestBench is to
help field engineers diagnose and repair problems faster. ``What they really want is a decision support system, one that supports both novice and expert,'' said the Carnegie Group's Bodin. ``They want a system that will help them through the diagnostic process; will augment their own intuition; will not constrain them in any way from arriving at the solution by forcing them down a specific path; and will fit the environment they have established, taking advantage of investments that they've already made in things like on-line documentation and graphics.''
Assessing Questor's Performance
How well has TestBench served Picker? Shortly after developing the initial version of Questor for its MTX family of products, Picker measured the system's effectiveness using two separate classes (each with approximately 18 field engineers) that had just graduated from Questor-based product training. The company had previously determined MTTD (mean-time-to-diagnose) for its manual diagnostic methods and used thi
s information as a baseline to assess Questor-supported repair of a set of six MTX system faults. In both classes, five of the six repairs were effected faster with Questor.
That means that one-sixth of the diagnoses were slower with Questor than when using paper-based resources and methods. Shouldn't every diagnosis go faster? It could be that with such a small sample base, one class attendee had solved that problem in the field the week before and so is lightning fast. It could be that talented, trained field engineers are just faster than expert systems. ``A `hot shot' with good intuition is going to be quicker than any codified process you can deliver,'' Grybush said.
Recognizing the limited value of these preliminary metrics, Grybush and his staff developed more effective methods of objectively assessing the diagnostic and repair performance improvements the Questor systems provide. ``We now categorize the class of objects that are serviced by people with these tools and can, over time, mak
e measurements of performance in sites where these tools were employed versus where they weren't,'' said Grybush.
This approach provides a broad sample base for drawing some statistically significant conclusions about diagnostic and repair performance. The idea is not to compare employee performance. ``The idea is to find out where people are spending most of their time, so we can concentrate on supporting those areas well,'' Grybush said.
When Picker decided to move forward with the Questor expert-system initiative, it had already identified significant benefits to be gained, but what wasn't known was their magnitude and value. ``We had a really hard time quantifying the benefits,'' said Picker's Larry Stanich, training program manager.
The fact that they couldn't quantify projected benefits gave Picker pause. ``We began with the very strong intuition that this would allow us to serve our customers better and faster and that this was going to provide us with a good deal of tangible benef
it in the long run,'' said David Kline, manager of magnetic resonance service engineering. ``We held strong to that assumption and built some of these tools. We did both an alpha and a beta test using training-center staff and known experts from the field. Then we deployed tools to a larger base of engineers in the field, and we used their feedback to improve the tool.''
As it turned out, determining the value of the benefits that were delivered was impossible as well. ``Our accounting system is not capable of capturing the benefits as we deploy these systems,'' explained Kline. ``We're now making up for that by creating accounting systems of our own, temporarily, to give us feedback.''
Field Assessment
How well does the on-line documentation support Questor? ``Suppose you're diagnosing an image-acquisition processor--that's a computer embedded in the diagnostic device,'' explained John Kuznicki, a Picker service-engineer specialist at one of its telephone-support resource centers.
``From the manual's table of contents, you position your mouse and click once to bring up the parts manual, which provides, for instance, a layout of the acquisition processor and all the power supplies. With a second click, you can see all the serviceable parts in the acquisition processor. Another click, and you can see a wave-form diagram. It's literally done in a matter of seconds. Compare this to hunting through paper documentation.'' Technical-support staff not only praise the result, on the whole, according to Stanich, they're 40 percent more productive.
Restructuring Field Service
The Challenge
Picker International wanted to maximize return on field-service investment by:
-- decreasing response time
-- lessening reliance on costly mainframes
-- increasing efficiency of field engineers
-- capturing and disseminating expert knowledge
The Response
These goals moved Picker to restructure its field-service operation by:
-- moving field engineers to a portable comp
uting platform
-- restructuring dispatching to get information to engineers more quickly
-- replacing static and bulky paper-based documentation with hypertext
documentation
The Focus
Expert-system technology let Picker tie together its field-service infrastructure by:
-- capturing site data for later inclusion in the knowledge base
-- giving on-site engineers access to all appropriate documentation
-- giving engineers access to the experiences of other engineers
-- providing practical information for design and manufacturing
Lessons Learned
The Picker experience demonstrates that:
-- Expert systems can be built incrementally. Don't wait for the final,
overarching vision to fall into place to get started and make progress.
-- The knowledge engineer should have some domain knowledge.
-- The object-oriented approach pays dividends in flexibility.
-- Capturing equipment and repair statistics enables the improvement not
only of support and repair processes, it enables the imp
rovement of the
products themselves.
-- Empirical information about product failures and repairs also helps in
creating products that are designed to be repaired.
-- Don't be surprised if your existing accounting structure can't provide
the information required; you may need to pilot a special accounting
approach.
-- Making information more broadly and flexibly available leverages that
information by enabling its use when and where it will do the most good.
Field-Service Laptop
Toshiba T4500 laptop: 486SL (25 MHz) with 8 MB of memory, an 85-MB hard drive (soon to be upgraded to 340 MB), a 14.4-Kbps fax modem, a trackball, and a monochrome display at 640- by 480-pixel resolution. Software includes DOS and Windows 3.1.
Field Link: Picker's Field Link system supports a variety of dispatching services, including call logging, service assignment, status reporting, and call closure.
Questor: Questor, which provides computerized support for pro
blem diagnosis and repair, starts with a Carnegie Group expert-system interface and incorporates knowledge and decision paths developed by Picker's technical-support and engineering staff.
Koan: This Picker-developed utility provides data-decompression and security features.
WorldView Press: From Interleaf, this is a hyperdocument ``reader'' that enables users to review product-specific documentation (including text, diagrams, figures, and schematics) on their laptops. Currently, three of Picker's diagnostic product lines are supported, with more on the way.
User Tools: At their discretion, users can mount word processors, spreadsheets, schedulers, E-mail tools, and other products for use on their laptops. Right now, no formal E-mail links exist between Picker's facility-based networks and the Field-Service laptops.
Illustration: TestBench Components
Illustration: Knowledge Editor Interface
Photograph: Michael Grybush, manager
of advanced services technology at Picker
Scott Wallace is a BYTE technical editor. You can reach him on the Internet or BIX at swallace @bix.com.
Experts in the Field
