Principles of Testing

Principles of Testing

Economics of Testing
It is both the driving force and the limiting factor
Driving - Earlier the errors are discovered and removed in the lifecycle, lower the cost of their removal
Limiting - Testing must end when the economic returns cease to make it worth while i.e the costs of testing process significantly outweigh the returns
Exhaustive Testing

•Testing every possible input over every possible output
• Can use every possible input condition as a test case

Is Exhaustive Testing feasible?

E.g a COBOL Compiler
• Impossible to create test cases to represent all valid cases.
• Impossible to create test cases for all invalid COBOL Programs
•The compiler has to be tested to see that it does not do what it is not supposed to do
E.g to successfully compile a syntactically incorrect program
•Exhaustive testing is hence impossible
•Implications: One cannot test a program completely to guarantee that it is error free
Economics
•Objective is to therefore find maximum errors with a finite number of test cases

Limitations of Software Testing

Even if we could generate the input, run the tests, and evaluate the output, we would not detect all faults:
•Correctness is not checked:
The programmer may have misinterpreted the specs, the specs may have misinterpreted the requirements
•There is no way to find missing paths due to coding errors

Psychology of Testing

•Test Engineers pursue defects not people
•Don’t assume that no error(s) will be found
•Test for Valid and Expected as well as Invalid and Unexpected
•The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section
•Testing is extremely creative and intellectually challenging

Test Case

•“A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.”
(…IEEE)
In other words, a planned sequence of actions (with the objective of finding errors)


A Good Test-Case
•Has a high probability of detecting error(s)
•Test cases help us discover information (.. Kaner)
e.g. of information objectives
• Help managers make ship / no-ship decisions.
• Minimize technical support costs.
• Assess conformance to specification.
• Minimize safety-related lawsuit risk.
• Verify correctness of the product.


Other Terminologies

Test Suite – A set of individual test cases/scenarios that are executed as a package, in a particular sequence and to test a particular aspect.
E.g. Test Suite for a GUI or Test Suite for functionality
Test Cycle – A test cycle consists of a series of test suites which comprises a complete execution set from the initial setup to the the test environment through reporting and clean up.
E.g. Integration test cycle / regression test cycle