Blackboxtesting's Blog

With that analogy in mind, let’s look at the ISTQB definition of software testing.
Let’s break the definition down into parts; the definition has some key phrases to remember. The definition starts with a description of testing as a process and then lists some objectives of the test process. First, let’s look at testing as a process:
• Process – Testing is a process rather than a single activity – there are a series of activities involved.
• All life cycle activities – looks at testing as a process that takes place throughout the software development life cycle. We saw earlier that the later in the life cycle we find bugs, the more expensive they are to fix. If we can find and fix requirements defects at the requirements stage, that must make commercial sense. We’ll build the right software, correctly and at a lower cost overall. So, the thought process of design ing tests early in the life cycle can help to prevent defects from being introduced into code. We sometimes refer to this as ‘verifying the test basis via the test design’. The test basis includes documents such as the requirements and design specifications.
• Both static and dynamic – that as well as tests where the software code is executed to demonstrate the results of running tests (often called dynamic testing) we can also test and find defects without exe cuting code. This is called static testing. This testing includes reviewing of documents (including source code) and static analysis. This is a useful and cost effective way of testing.
• Planning - Activities take place before and after test execution. We need to manage the testing; for example, we plan what we want to do; we control the test activities; we report on testing progress and the status of the software under test; and we finalize or close testing when a phase completes.
• Preparation - We need to choose what testing we’ll do, by selecting test con ditions and designing test cases. .
• Evaluation – As well as executing the tests, we must check the results and evaluate the software under test and the completion criteria, which help us decide whether we have finished testing and whether the software product has passed the tests.
• Software products and related work products – We don’t just test code. We test the requirements and design specifications, and we test related documents such as operation, user and training material. Static and dynamic testing are both needed to cover the range of products we need to test.
The second part of the definition covers the some of the objectives for testing -the reasons why we do it:
• Determine that (software products) satisfy specified requirements - Some of the testing we do is focused on checking products against the specification for the product; for example we review the design to see if it meets require ments, and then we might execute the code to check that it meets the design. If the product meets its specification, we can provide that information to help stakeholders judge the quality of the product and decide whether it is ready for use.
• Demonstrate that (software products) are fit for purpose – This is slightly different to the point above; after all the specified requirements might be wrong or incomplete. ‘Fit for purpose’ looks at whether the software does enough to help the users to carry out their tasks; we look at whether the soft ware does what the user might reasonably expect. For example, we might look at who might purchase or use the software, and check that it does do what they expect; this might lead us to add a review of the marketing mate rial to our static tests, to check that expectations of the software are properly set. One way of judging the quality of a product is by how fit it is for its purpose.
• Detect defects - We most often think of software testing as a means of detecting faults or defects that in operational use will cause failures. Finding the defects helps us understand the risks associated with putting the software into operational use, and fixing the defects improves the quality of the prod ucts. However, identifying defects has another benefit. With root cause analysis, they also help us improve the development processes and make fewer mistakes in future work.
This is a suitable definition of testing for any level of testing, from compo-nent testing through to acceptance testing, provided that we remember to take the varying objectives of these different levels of testing into account.
We can clearly see now why the common perception of testing (that it only consists of running tests, i.e. executing the software) is not complete. This is one of the testing activities, but not all of the testing process.

We’ve seen that testing helps us find defects and improve software quality. How much testing should we do? We have a choice: test everything, test nothing or test some of the software. Now, your immediate response to that may well be to say, ‘Everything must be tested’. We don’t want to use software that has not been completely tested, do we? This implies that we must exercise every aspect of a software system during testing. What we need to consider is whether we must, or even can, test completely.
Let’s look at how much testing we’d need to do to be able to test exhaus-tively. How many tests would you need to do to completely test a one-digit numeric field? The immediate question is, ‘What you mean by test completely?’ There are 10 possible valid numeric values but as well as the valid values we need to ensure that all the invalid values are rejected. There are 26 uppercase alpha characters, 26 lower case, at least 6 special and punctuation characters as well as a blank value. So there would be at least 68 tests for this example of a one-digit field.
This problem just gets worse as we look at more realistic examples. In prac-tice, systems have more than one input field with the fields being of varying sizes. These tests would be alongside others such as running the tests in differ-
Section 2 What is testing? 11
ent environments. If we take an example where one screen has 15 input fields, each having 5 possible values, then to test all of the valid input value combina-tions you would need 30 517 578 125 (515) tests! It is unlikely that the project timescales would allow for this number of tests.
Testing our one-digit field with values 2, 3 and 4 makes our tests more thor-ough, but it does not give us more information than if we had just tested with the value 3.
Pressures on a project include time and budget as well as pressure to deliver a technical solution that meets the customers’ needs. Customers and project managers will want to spend an amount on testing that provides a return on investment for them. This return on investment includes prevent-ing failures after release that are costly. Testing completely – even if that is what customers and project managers ask for – is simply not what they can afford.
Instead we need a test approach which provides the right amount of testing for this project, these customers (and other stakeholders) and this software. We do this by aligning the testing we do with the risks for the customers, the stake-holders, the project and the software. Assessing and managing risk is one of the most important activities in any project, and is a key activity and reason for testing. Deciding how much testing is enough should take account of the level of risk, including technical and business risks related to the product and project constraints such as time and budget.
We carry out a risk assessment to decide how much testing to do. We can then vary the testing effort based on the level of risk in different areas. Additionally, testing should provide sufficient information to stakeholders to make informed decisions about the release of the software or system we’re testing, for the next development step or handover to customers. The effort put into the quality assurance and testing activities needs to be tai lored to the risks and costs associated with the project. Because of the limits in the budget, the time, and in testing we need to decide how we will focus our testing, based on the risks.

Why is it that software systems sometimes don’t work correctly? We know that people make mistakes – we are fallible.
If someone makes an error or mistake in using the software, this may lead directly to a problem – the software is used incorrectly and so does not behave as we expected. However, people also design and build the software and they can make mistakes during the design and build. These mistakes mean that there are flaws in the software itself. These are called defects or sometimes bugs or faults. Remember, the software is not just the code; check the definition of soft-ware again to remind yourself.
When the software code has been built, it is executed and then any defects may cause the system to fail to do what it should do (or do something it shouldn’t), causing a failure. Not all defects result in failures; some stay dormant in the code and we may never notice them.
Do our mistakes matter?
Let’s think about the consequences of mistakes. We agree that any human
being, programmers and testers included, can make an error. These errors may
produce defects in the software code or system, or in a document. If a defect in
code is executed, the system may experience a failure. So the mistakes we make
matter partly because they have consequences for the products for which we are
responsible.
Our mistakes are also important because software systems and projects are complicated. Many interim and final products are built during a project, and people will almost certainly make mistakes and errors in all the activities of the build. Some of these are found and removed by the authors of the work, but it is difficult for people to find their own mistakes while building a product. Defects in software, systems or documents may result in failures, but not all defects do cause failures. We could argue that if a mistake does not lead to a defect or a defect does not lead to a failure, then it is not of any importance -we may not even know we’ve made an error.
Our fallibility is compounded when we lack experience, don’t have the right information, misunderstand, or if we are careless, tired or under time pressure. All these factors affect our ability to make sensible decisions – our brains either don’t have the information or cannot process it quickly enough.
Additionally, we are more likely to make errors when dealing with perplex-ing technical or business problems, complex business processes, code or infra-structure, changing technologies, or many system interactions. This is because our brains can only deal with a reasonable amount of complexity or change -when asked to deal with more our brains may not process the information we have correctly.
It is not just defects that give rise to failure. Failures can also be caused by environmental conditions as well: for example, a radiation burst, a strong mag-netic field, electronic fields, or pollution could cause faults in hardware or firmware. Those faults might prevent or change the execution of software. Failures may also arise because of human error in interacting with the software, perhaps a wrong input value being entered or an output being misinterpreted. Finally, failures may also be caused by someone deliberately trying to cause a failure in a system – malicious damage.
When we think about what might go wrong we have to consider defects and failures arising from:
• errors in the specification, design and implementation of the software and system;
• errors in use of the system;
• environmental conditions;
• intentional damage;
• potential consequences of earlier errors, intentional damage, defects and failures.

In this chapter, we will introduce you to the fundamentals of testing: why testing is needed; its limitations, objectives and purpose; the principles behind testing; the process that testers follow; and some of the psychological factors that testers must consider in their work. By reading this chapter you’ll gain an understanding of the fundamentals of testing and be able to describe those fundamentals.

WHY IS TESTING NECESSARY?

6 Explain the fundamental principles in testing. (K2)

Introduction In this section, we’re going to kick off the book with a discussion on why testing matters. We’ll describe and illustrate how software defects or bugs can cause problems for people, the environment or a company. We’ll draw important dis-tinctions between defects, their root causes and their effects. We’ll explain why testing is necessary to find these defects, how testing promotes quality, and how testing fits into quality assurance. In this section, we will also introduce some fundamental principles of testing. As we go through this section, watch for the Syllabus terms bug, defect, error, failure, fault, mistake, quality, risk, software, testing and exhaustive testing. You’ll find these terms defined in the glossary.

You may be asking ‘what is testing?’ and we’ll look more closely at the defi-nition of testing in Section 1.2. For the moment, let’s adopt a simple everyday-life usage: ‘when we are testing something we are checking whether it is OK’. We’ll need to refine that when we define software testing later on. Let’s start by considering why testing is needed. Testing is necessary because we all make mis-takes. Some of those mistakes are unimportant, but some of them are expensive or dangerous. We need to check everything and anything we produce because things can always go wrong – humans make mistakes all the time – it is what we do best!
Because we should assume our work contains mistakes, we all need to check our own work. However, some mistakes come from bad assumptions and blind spots, so we might make the same mistakes when we check our own work as we made when we did it. So we may not notice the flaws in what we have done. Ideally, we should get someone else to check our work – another person is more likely to spot the flaws.
In this book, we’ll explore the implications of these two simple paragraphs again and again. Does it matter if there are mistakes in what we do? Does it matter if we don’t find some of those flaws? We know that in ordinary life, some of our mistakes do not matter, and some are very important. It is the same with software systems. We need to know whether a particular error is likely to cause problems. To help us think about this, we need to consider the context within which we use different software systems.