Beautiful Testing. Edited by Tim Riley and Adam Goucher. Beijing Cambridge Farnham Köln Sebastopol Taipei Tokyo - PDF

Beautiful Testing Edited by Tim Riley and Adam Goucher Beijing Cambridge Farnham Köln Sebastopol Taipei Tokyo Beautiful Testing Edited by Tim Riley and Adam Goucher Copyright 2010 O Reilly Media, Inc..

Please download to get full document.

View again

of 21
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.

Public Notices

Publish on:

Views: 28 | Pages: 21

Extension: PDF | Download: 0

Beautiful Testing Edited by Tim Riley and Adam Goucher Beijing Cambridge Farnham Köln Sebastopol Taipei Tokyo Beautiful Testing Edited by Tim Riley and Adam Goucher Copyright 2010 O Reilly Media, Inc.. All rights reserved. Printed in the United States of America. Published by O Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA O Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles ( For more information, contact our corporate/ institutional sales department: or Editor: Mary E. Treseler Production Editor: Sarah Schneider Copyeditor: Genevieve d Entremont Proofreader: Sarah Schneider Indexer: John Bickelhaupt Cover Designer: Mark Paglietti Interior Designer: David Futato Illustrator: Robert Romano Printing History: October 2009: First Edition. O Reilly and the O Reilly logo are registered trademarks of O Reilly Media, Inc. Beautiful Testing, the image of a beetle, and related trade dress are trademarks of O Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: [V] All royalties from this book will be donated to the UN Foundation s Nothing But Nets campaign to save lives by preventing malaria, a disease that kills millions of children in Africa each year. C O N T E N T S PREFACE by Adam Goucher xiii Part One BEAUTIFUL TESTERS 1 WAS IT GOOD FOR YOU? 3 by Linda Wilkinson 2 BEAUTIFUL TESTING SATISFIES STAKEHOLDERS 15 by Rex Black For Whom Do We Test? 16 What Satisfies? 18 What Beauty Is External? 20 What Beauty Is Internal? 23 Conclusions 25 3 BUILDING OPEN SOURCE QA COMMUNITIES 27 by Martin Schröder and Clint Talbert Communication 27 Volunteers 28 Coordination 29 Events 32 Conclusions 35 4 COLLABORATION IS THE CORNERSTONE OF BEAUTIFUL PERFORMANCE TESTING 37 by Scott Barber Setting the Stage %?!? Fail 38 The Memory Leak That Wasn t 45 Can t Handle the Load? Change the UI 46 It Can t Be the Network 48 Wrap-Up 51 Part Two BEAUTIFUL PROCESS 5 JUST PEACHY: MAKING OFFICE SOFTWARE MORE RELIABLE WITH FUZZ TESTING 55 by Kamran Khan User Expectations 55 What Is Fuzzing? 57 Why Fuzz Test? 57 vii Fuzz Testing 60 Future Considerations 65 6 BUG MANAGEMENT AND TEST CASE EFFECTIVENESS 67 by Emily Chen and Brian Nitz Bug Management 68 The First Step in Managing a Defect Is Defining It 70 Test Case Effectiveness 77 Case Study of the OpenSolaris Desktop Team 79 Conclusions 83 Acknowledgments 83 References 84 7 BEAUTIFUL XMPP TESTING 85 by Remko Tronçon Introduction 85 XMPP Testing XMPP Protocols 88 Unit Testing Simple Request-Response Protocols 89 Unit Testing Multistage Protocols 94 Testing Session Initialization 97 Automated Interoperability Testing 99 Diamond in the Rough: Testing XML Validity 101 Conclusions 101 References BEAUTIFUL LARGE-SCALE TEST AUTOMATION 103 by Alan Page Before We Start 104 What Is Large-Scale Test Automation? 104 The First Steps 106 Automated Tests and Test Case Management 107 The Automated Test Lab 111 Test Distribution 112 Failure Analysis 114 Reporting 114 Putting It All Together BEAUTIFUL IS BETTER THAN UGLY 119 by Neal Norwitz, Michelle Levesque, and Jeffrey Yasskin The Value of Stability 120 Ensuring Correctness 121 Conclusions TESTING A RANDOM NUMBER GENERATOR 129 by John D. Cook What Makes Random Number Generators Subtle to Test? 130 Uniform Random Number Generators 131 viii C O N T E N T S Nonuniform Random Number Generators 132 A Progression of Tests 134 Conclusions CHANGE-CENTRIC TESTING 143 by Murali Nandigama How to Set Up the Document-Driven, Change-Centric Testing Framework? 145 Change-Centric Testing for Complex Code Development Models 146 What Have We Learned So Far? 152 Conclusions SOFTWARE IN USE 155 by Karen N. Johnson A Connection to My Work 156 From the Inside 157 Adding Different Perspectives 159 Exploratory, Ad-Hoc, and Scripted Testing 161 Multiuser Testing 163 The Science Lab 165 Simulating Real Use 166 Testing in the Regulated World 168 At the End SOFTWARE DEVELOPMENT IS A CREATIVE PROCESS 171 by Chris McMahon Agile Development As Performance 172 Practice, Rehearse, Perform 173 Evaluating the Ineffable 174 Two Critical Tools 174 Software Testing Movements 176 The Beauty of Agile Testing 177 QA Is Not Evil 178 Beauty Is the Nature of This Work 179 References TEST-DRIVEN DEVELOPMENT: DRIVING NEW STANDARDS OF BEAUTY 181 by Jennitta Andrea Beauty As Proportion and Balance 181 Agile: A New Proportion and Balance 182 Test-Driven Development 182 Examples Versus Tests 184 Readable Examples 185 Permanent Requirement Artifacts 186 Testable Designs 187 Tool Support 189 Team Collaboration 192 Experience the Beauty of TDD 193 References 194 C O N T E N T S ix 15 BEAUTIFUL TESTING AS THE CORNERSTONE OF BUSINESS SUCCESS 195 by Lisa Crispin The Whole-Team Approach 197 Automating Tests 199 Driving Development with Tests 202 Delivering Value 206 A Success Story 208 Post Script PEELING THE GLASS ONION AT SOCIALTEXT 209 by Matthew Heusser It s Not Business It s Personal 209 Tester Remains On-Stage; Enter Beauty, Stage Right 210 Come Walk with Me, The Best Is Yet to Be 213 Automated Testing Isn t 214 Into Socialtext 215 A Balanced Breakfast Approach 227 Regression and Process Improvement 231 The Last Pieces of the Puzzle 231 Acknowledgments BEAUTIFUL TESTING IS EFFICIENT TESTING 235 by Adam Goucher SLIME 235 Scripting 239 Discovering Developer Notes 240 Oracles and Test Data Generation 241 Mindmaps 242 Efficiency Achieved 244 Part Three BEAUTIFUL TOOLS 18 SEEDING BUGS TO FIND BUGS: BEAUTIFUL MUTATION TESTING 247 by Andreas Zeller and David Schuler Assessing Test Suite Quality 247 Watching the Watchmen 249 An AspectJ Example 252 Equivalent Mutants 253 Focusing on Impact 254 The Javalanche Framework 255 Odds and Ends 255 Acknowledgments 256 References REFERENCE TESTING AS BEAUTIFUL TESTING 257 by Clint Talbert Reference Test Structure 258 x C O N T E N T S Reference Test Extensibility 261 Building Community CLAM ANTI-VIRUS: TESTING OPEN SOURCE WITH OPEN TOOLS 269 by Tomasz Kojm The Clam Anti-Virus Project 270 Testing Methods 270 Summary 283 Credits WEB APPLICATION TESTING WITH WINDMILL 285 by Adam Christian Introduction 285 Overview 286 Writing Tests 286 The Project 292 Comparison 293 Conclusions 293 References TESTING ONE MILLION WEB PAGES 295 by Tim Riley In the Beginning 296 The Tools Merge and Evolve 297 The Nitty-Gritty 299 Summary 301 Acknowledgments TESTING NETWORK SERVICES IN MULTIMACHINE SCENARIOS 303 by Isaac Clerencia The Need for an Advanced Testing Tool in ebox 303 Development of ANSTE to Improve the ebox QA Process 304 How ebox Uses ANSTE 307 How Other Projects Can Benefit from ANSTE 315 A CONTRIBUTORS 317 INDEX 323 C O N T E N T S xi Preface I DON T THINK BEAUTIFUL TESTING COULD HAVE BEEN PROPOSED, much less published, when I started my career a decade ago. Testing departments were unglamorous places, only slightly higher on the corporate hierarchy than front-line support, and filled with unhappy drones doing rote executions of canned tests. There were glimmers of beauty out there, though. Once you start seeing the glimmers, you can t help but seek out more of them. Follow the trail long enough and you will find yourself doing testing that is: Fun Challenging Engaging Experiential Thoughtful Valuable Or, put another way, beautiful. Testing as a recognized practice has, I think, become a lot more beautiful as well. This is partly due to the influence of ideas such as test-driven development (TDD), agile, and craftsmanship, but also the types of applications being developed now. As the products we develop and the xiii ways in which we develop them become more social and less robotic, there is a realization that testing them doesn t have to be robotic, or ugly. Of course, beauty is in the eye of the beholder. So how did we choose content for Beautiful Testing if everyone has a different idea of beauty? Early on we decided that we didn t want to create just another book of dry case studies. We wanted the chapters to provide a peek into the contributors views of beauty and testing. Beautiful Testing is a collection of chapter-length essays by over 20 people: some testers, some developers, some who do both. Each contributor understands and approaches the idea of beautiful testing differently, as their ideas are evolving based on the inputs of their previous and current environments. Each contributor also waived any royalties for their work. Instead, all profits from Beautiful Testing will be donated to the UN Foundation s Nothing But Nets campaign. For every $10 in donations, a mosquito net is purchased to protect people in Africa against the scourge of malaria. Helping to prevent the almost one million deaths attributed to the disease, the large majority of whom are children under 5, is in itself a Beautiful Act. Tim and I are both very grateful for the time and effort everyone put into their chapters in order to make this happen. How This Book Is Organized While waiting for chapters to trickle in, we were afraid we would end up with different versions of this is how you test or keep the bar green. Much to our relief, we ended up with a diverse mixture. Manifestos, detailed case studies, touching experience reports, and war stories from the trenches Beautiful Testing has a bit of each. The chapters themselves almost seemed to organize themselves naturally into sections. Part I, Beautiful Testers Testing is an inherently human activity; someone needs to think of the test cases to be automated, and even those tests can t think, feel, or get frustrated. Beautiful Testing therefore starts with the human aspects of testing, whether it is the testers themselves or the interactions of testers with the wider world. Chapter 1, Was It Good for You? Linda Wilkinson brings her unique perspective on the tester s psyche. Chapter 2, Beautiful Testing Satisfies Stakeholders Rex Black has been satisfying stakeholders for 25 years. He explains how that is beautiful. Chapter 3, Building Open Source QA Communities Open source projects live and die by their supporting communities. Clint Talbert and Martin Schröder share their experiences building a beautiful community of testers. xiv P R E F A C E Chapter 4, Collaboration Is the Cornerstone of Beautiful Performance Testing Think performance testing is all about measuring speed? Scott Barber explains why, above everything else, beautiful performance testing needs to be collaborative. Part II, Beautiful Process We then progress to the largest section, which is about the testing process. Chapters here give a peek at what the test group is doing and, more importantly, why. Chapter 5, Just Peachy: Making Office Software More Reliable with Fuzz Testing To Kamran Khan, beauty in office suites is in hiding the complexity. Fuzzing is a test technique that follows that same pattern. Chapter 6, Bug Management and Test Case Effectiveness Brian Nitz and Emily Chen believe that how you track your test cases and bugs can be beautiful. They use their experience with OpenSolaris to illustrate this. Chapter 7, Beautiful XMPP Testing Remko Tronçon is deeply involved in the XMPP community. In this chapter, he explains how the XMPP protocols are tested and describes their evolution from ugly to beautiful. Chapter 8, Beautiful Large-Scale Test Automation Working at Microsoft, Alan Page knows a thing or two about large-scale test automation. He shares some of his secrets to making it beautiful. Chapter 9, Beautiful Is Better Than Ugly Beauty has always been central to the development of Python. Neal Noritz, Michelle Levesque, and Jeffrey Yasskin point out that one aspect of beauty for a programming language is stability, and that achieving it requires some beautiful testing. Chapter 10, Testing a Random Number Generator John D. Cook is a mathematician and applies a classic definition of beauty, one based on complexity and unity, to testing random number generators. Chapter 11, Change-Centric Testing Testing code that has not changed is neither efficient nor beautiful, says Murali Nandigama; however, change-centric testing is. Chapter 12, Software in Use Karen N. Johnson shares how she tested a piece of medical software that has had a direct impact on her nonwork life. Chapter 13, Software Development Is a Creative Process Chris McMahon was a professional musician before coming to testing. It is not surprising, then, that he thinks beautiful testing has more to do with jazz bands than manufacturing organizations. Chapter 14, Test-Driven Development: Driving New Standards of Beauty Jennitta Andrea shows how TDD can act as a catalyst for beauty in software projects. P R E F A C E xv Chapter 15, Beautiful Testing As the Cornerstone of Business Success Lisa Crispin discusses how a team s commitment to testing is beautiful, and how that can be a key driver of business success. Chapter 16, Peeling the Glass Onion at Socialtext Matthew Heusser has worked at a number of different companies in his career, but in this chapter we see why he thinks his current employer s process is not just good, but beautiful. Chapter 17, Beautiful Testing Is Efficient Testing Beautiful testing has minimal retesting effort, says Adam Goucher. He shares three techniques for how to reduce it. Part III, Beautiful Tools Beautiful Testing concludes with a final section on the tools that help testers do their jobs more effectively. Chapter 18, Seeding Bugs to Find Bugs: Beautiful Mutation Testing Trust is a facet of beauty. The implication is that if you can t trust your test suite, then your testing can t be beautiful. Andreas Zeller and David Schuler explain how you can seed artificial bugs into your product to gain trust in your testing. Chapter 19, Reference Testing As Beautiful Testing Clint Talbert shows how Mozilla is rethinking its automated regression suite as a tool for anticipatory and forward-looking testing rather than just regression. Chapter 20, Clam Anti-Virus: Testing Open Source with Open Tools Tomasz Kojm discusses how the ClamAV team chooses and uses different testing tools, and how the embodiment of the KISS principle is beautiful when it comes to testing. Chapter 21, Web Application Testing with Windmill Adam Christian gives readers an introduction to the Windmill project and explains how even though individual aspects of web automation are not beautiful, their combination is. Chapter 22, Testing One Million Web Pages Tim Riley sees beauty in the evolution and growth of a test tool that started as something simple and is now anything but. Chapter 23, Testing Network Services in Multimachine Scenarios When trying for 100% test automation, the involvement of multiple machines for a single scenario can add complexity and non-beauty. Isaac Clerencia showcases ANSTE and explains how it can increase beauty in this type of testing. Beautiful Testers following a Beautiful Process, assisted by Beautiful Tools, makes for Beautiful Testing. Or at least we think so. We hope you do as well. xvi P R E F A C E Using Code Examples This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: Beautiful Testing, edited by Tim Riley and Adam Goucher. Copyright 2010 O Reilly Media, Inc., If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at Safari Books Online Safari Books Online is an on-demand digital library that lets you easily search over 7,500 technology and creative reference books and videos to find the answers you need quickly. With a subscription, you can read any page and watch any video from our library online. Read books on your cell phone and mobile devices. Access new titles before they are available for print, and get exclusive access to manuscripts in development and post feedback for the authors. Copy and paste code samples, organize your favorites, download chapters, bookmark key sections, create notes, print out pages, and benefit from tons of other time-saving features. O Reilly Media has uploaded this book to the Safari Books Online service. To have full digital access to this book and others on similar topics from O Reilly and other publishers, sign up for free at How to Contact Us Please address comments and questions concerning this book to the publisher: O Reilly Media, Inc Gravenstein Highway North Sebastopol, CA (in the United States or Canada) (international or local) (fax) P R E F A C E xvii We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at: To comment or ask technical questions about this book, send to: For more information about our books, conferences, Resource Centers, and the O Reilly Network, see our website at: Acknowledgments We would like to thank the following people for helping make Beautiful Testing happen: Dr. Greg Wilson. If he had not written Beautiful Code, we would never have had the idea nor a publisher for Beautiful Testing. All the contributors who spent many hours writing, rewriting, and sometimes rewriting again their chapters, knowing that they will get nothing in return but the satisfaction of helping prevent the spread of malaria. Our technical reviewers: Kent Beck, Michael Feathers, Paul Carvalho, and Gary Pollice. Giving useful feedback is sometimes as hard as receiving it, but what we got from them certainly made this book more beautiful. And, of course, our wives and children, who put up with us doing book stuff over the last year. Adam Goucher xviii P R E F A C E C H A P T E R T E N Testing a Random Number Generator John D. Cook ACCORDING TO THE CLASSICAL DEFINITION OF BEAUTY, something is beautiful if it exhibits both complexity and unity. Professor Gene Veith explained this idea in an editorial by describing two kind of paintings: * In painting a black canvas has unity, but it has no complexity. A canvas of random paint splatterings has complexity, but it has no unity. Michelangelo s painting of the Sistine Chapel ceiling has rich detail along with order and balance. It exhibits complexity and unity. It is beautiful. Some works of beauty are easy to appreciate because both the complexity and the unity are apparent. I would say the Sistine Chapel falls in this category. However, other works require more education to appreciate because it takes knowledge and skill to see either the complexity or the unity. Modern jazz may fall into this latter category. The complexity is obvious, but the unity may not be apparent to untrained ears. Tests for random number generators may be more like modern jazz than the Sistine Chapel; the complexity is easier to see than the unity. But with some introduction, the unity can be appreciated. * Veith, Gene Edward. Acquired taste, World Magazine. February 29, What Makes Random Number
Related Search
Similar documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks