http://www.spinellis.gr/pubs/Breview/2001-IEEESW-Testing/html/review.html
This is an HTML rendering of a working paper draft that led to a publication. The publication should always be cited in preference to this draft using the following reference:

This document is also available in PDF format.

The document's metadata is available in BibTeX format.

Find the publication on Google Scholar

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Diomidis Spinellis Publications

© 2001 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

Practical Testing Advice

Diomidis Spinellis

Testing Computer Software, second ed. by Cem Kaner, Jack Falk, and Hung Quoc Nguyen, John Wiley & Sons, New York, 1999, ISBN 0-471-35846-0, 480 pp., US$34.99.

Testing is sometimes regarded as the ugly duckling of software development. Programmers want to write perfect code, development managers prefer to schedule releases without the additional uncertainties testing results bring, and financial officers surely want to do away with additional testing costs. Fortunately, three authors who love this ugly duckling wrote Testing Computer Software, and they successfully transfer their knowledge and-more importantly-their views to the reader.

The authors primarily target testers and test managers involved in producing non safety-critical software. They readily admit that computer science graduates rarely have useful testing knowledge-software testing can be better performed by suitably trained college graduates. The authors thus painstakingly introduce all relevant software engineering and programming concepts needed to perform testing, and they also point to relevant material for further reading. The software life cycle, corresponding testing cycles, glass-box versus black-box testing, regression testing, maintenance, and boundary conditions are some of the topics covered.

Bug tracking

An important aspect of testing is the clear and effective communication between testers, developers, and managers-a key factor being comprehensive and well-organized problem reports. Problem reports should clearly identify the program and the problem, the problem type, its severity, and steps to reproduce it. In addition, the development team should further expand such reports by grouping them into functional categories, assigning problems to specific developers, and tracking their evolution and final resolution.

Having designed and implemented a bug tracking system for managing a medium-scale development project, I was impressed by the pragmatic attitude the authors advanced; all suggestions in this area are easy to put into practice and are clearly the result of extensive real-life experience. The implementation of a bug-tracking system, which is outlined in a separate chapter, is likely to bring forward political issues. Although many useful metrics can be extracted from such a system, the authors wisely point out that it would be counterproductive to use it for tracking individual performance of programmers or testers. Reports can still help make estimates based on past performance figures and judging project progress.

Another particularly thorny issue concerns the problem report life cycle. A state diagram would help readers better understand the temporal and organizational relationship between problem resolution and status. However, even the authors appear split on the issue of how to treat deferred bugs.

Test cases

Designing effective test cases is a challenge for black-box testers. Although experience is clearly an important asset, the numerous methods and heuristics described provide a starting point for a junior tester. Equivalence classes, boundary values, and regression testing can minimize the number and increase the effectiveness of test cases. Understanding state transitions, race conditions, time dependencies, and the judicious use of random input will help uncover more subtle problems. User manuals and help systems are also part of the tester's work domain; the authors devote a separate chapter to this area.

Although the book's second edition was published in 1999, the material concerning the testing of printers, testing tools, localization, and legal aspects needs to be brought up-to-date. Most printers nowadays are not directly driven by end-user software and are seldom operated using simple escape sequences. Testing tools suitable for character input/output are not applicable to graphical user interfaces, character code pages are disappearing in favor of Unicode, and language libraries and operating systems provide internationalization support with radically different testing demands from earlier ad hoc localization approaches. References also cover only material published up to 1991.

Test management

Test management, planning, and documentation are important aspects of any large testing effort. The relevant IEEE standards offer important guidance; the authors describe in simple terms the contents and usefulness of test plan elements and related documents. A concrete example for each document would have been useful but would have significantly increased the book's 480 pages. Test plans inevitably must organize the numerous elements to be tested into coherent and understandable groups. We can use lists, tables, outlines, and matrices to organize this information; the book's examples for each type of chart help identify which is most appropriate.

The authors devote the last part of the book to testing management, explaining why we should integrate it into the software development life cycle. Important milestones include alpha and beta testing and the user interface freeze. Testing can be performed in-house by a dedicated testing group, or it can be outsourced to an external agency; readers who have read the previous book chapters will probably favor the former approach. The authors' empathy with software testers and managers is commendable. Recognizing that testing is often an entry-level job that can lead to a software development position can help testers work toward professional advancement and help managers constructively deal with the inevitably high staff turnover rate.

Ready for the third edition

I enjoyed reading Testing Computer Software. The text contains numerous highlights offering practical advice, authoritative figures you can cite to customers and higher management, and entertaining anecdotes to share with coworkers. Although some sections need updating, I still think it is a valuable training and reference source for software testers, managers, and developers.

Diomidis Spinellis is an assistant professor in the Department of Management Science and Technology at the Athens University of Economics and Business. Contact him at dds@aueb.gr.