Test Automation & Tools

Test Automation & Tools

•What is test automation
•Why Automate
•Test Cases suitable for automation
•Test Cases unsuitable for automation
•Automation Strategies
•Pros & Cons - Manual Vs. Automated testing
•Let us get to know some Automation Tools
•Summary

What is Test Automation?

Replace manual Test Engineers by automating everything

Automated testing by itself will ensure high quality

Challenges of Testing

•Today we witness huge and very complex code churned out at faster pace
•Testing is struggling to keep pace with Technological obsolescence and Advancements
•Fast Changing requirements
•Waterfall Model of development leaves restrained scope for optimal testing
•Testing is often squeezed to the fag-end of the SDLC

How to overcome the challenges?

•Trade coverage for risk
Reduces scope
Shortens test cycles
•Outsource Testing work
•Introduce/ Enhance AUTOMATION

Why Automate?

•Repeatability : Tests can be executed multiple times and consistently each time.
• Leverage : Executing tests that were never performed manually at all.
• Accumulation : The number of test required for coverage increases with each enhancement done to applications

Benefits of Automation

•Improves human productivity
Broader test coverage
•Reduces redundancy
Automate repetitive tasks
•Enables process improvements
Increase regression testing
Expose risk areas
Better management of testing (with Test-Director etc.)

Automation MYTHS

•Automate everything
•Shorten the test cycle by 50%
•Automation is ‘Easy’
•An automated test is faster, because it needs no human intervention
•Automation will lead to "significant labor cost savings"

Suitable candidates for Automation

þTests that need to be run for every build of the application (sanity check, regression test)
þTests that use multiple data values for the same actions (data-driven tests)
þTests that require detailed information from application internals (e.g., SQL, GUI attributes)
þStress/load testing

þTest cases that can not be done manually
þScenario/ Use-Case / Transaction Flow test
þConfiguration test
þMulti-user Scenarios

Examples :

•While testing the MS word we can check if we open a new document 1000 times, default name should be “Doc1000”
•In the case of adding more number of data say for example 1lakh of data to the database
•When broken links or Orphanage links are to be identified from the web application
•While testing the MS word we can make different styles by using automation and can check

•After preparing the next build we can automate to test that build contains all the files and folders
•Performance testing of the server, network application at various anticipated load levels (if 100 users are logging to yahoo mail, we can find out the response time and behavior of the site)
•GUI properties like Size, Position, Label, Visible, Enable, Value etc. to be tested.(Customer requirement is that the company logo should be displayed at the top center in all the screens)

Unsuitable to Automate

ýAesthetics (ease of use, usability, look and feel)
ý One-time testing
ý Start testing immediately
ý Ad-hoc/random or Exploratory testing
ý Tests without predictable results
ý Testing dynamic areas under development

Why Automation Fails?

•Not clear or enough Requirements
•Staff skills may not match needs
•Staff turnover
•Insufficient time

Strategies for Automation

•Don’t try to automate everything right away
•Focus on the most important test cases first
Regression tests
High priority features or product requirements
High risk use cases, capabilities
•Start with repetitive tasks susceptible to human error if done manually

Manual Testing Pros & Cons

pros
•Quick & inexpensive
•Simulates real-world use better
•Better for extensive analysis of test cases as well as results
•Best when test steps and results are not well defined
•Can explore new test cases
•Does not require technically trained staff

cons
•Might not be consistent in re-running test cases.
•Re-running large volumes of test cases is expensive and tedious.
•Work-force turnover may upset plans

Automated Testing Pros & Cons

pros

•Can execute test cases unattended.
•Can cost-effectively run large volumes of test cases.
•Can cost-effectively re-run large volumes of test cases repetitively

cons

•Time-consuming & expensive
•Maintaining (modifications etc.) of test cases / scripts are difficult and expensive
•Requires a highly trained technical staff
•Re-running old tests may not necessarily spot new bugs

What Is a Test Script?

A series of commands or events stored in a script language file that execute a test case and report the results.

Fundamentals of Test Automation

•Maintainability
Applications are maintained continuously
Cross-reference test scripts o the application
Design to avoid regression
Adopt a test framework
•Optimization
Enough tests to do the job without having to manage to many

•Independence ( test cases)
Independent data
Independent context
Result independence
•Modularity (test scripts)
Identify common scripts

Synchronization
• Execute the test scrip and the application at the same pace
• Different conditions may exist at the time of playback than existed when the test was created, precise timing coincidence may not be possible

Test Framework

•Test framework is like an application architecture
•It outlines the overall structure for the automated test environment, defines common functions, standard tests, provides templates for test structure, naming convention
•Leading to a maintainable and transferable test library

•Common Functions (e.g. Setup, Sign on, Monitor, Log Error, Recover, Signoff, Cleanup)
•Standard Tests (e.g. Walk thru to navigate thru each menu item)
•Test Script templates – (Header : Test description, inputs, outputs, special requirement, Dependencies, Common routines like setup etc.,End))
•Test Dictionary –( Vocabulary, Naming Conventions, Cross reference names to application)

Test Automation Approach

Approach / Profile


Capture /Playback Application already in test phase or maintenance
Stable application.
Variable Capture/ Variable Playback
Application in planning, analysis or design
Unstable or stable application
Technical test team

Different type of tools

•Reviews and Inspection
•Test planning
•Test design & Development
•Test execution & verification
•Test support

Automation tools are not one-size-fits-all

Tools for Reviews & Inspection

•Requirement analysis
Compuware’s Reconcile/TeleLogic’s DOORS/Rational Requisite Pro
•Complexity analysis
Compuware’s Numega
•Code comprehension
Compuware’s Numega
•Syntax analysis
Lint for C and Jlint for Java

Tools for test planning

•Templates for test plan documentations
•Test schedule and staffing estimates
•Complexity analyzer

Tools for Test Design & Development

•Test data generator
Compuware’s File-Aid
•Requirements-based test design tool
•Capture/Playback

Test Execution and Evaluation Tools

•Capture/playback
Mercury’s WinRunner
Compuware’s QARun
Segue’s Silk Test
Rational’s Visual Test
Rational’s ROBOT
•Coverage analysis
•Memory testing

•Test case management
Mercury’s Test Director
Compuware’s QADirector
•Simulators & performance
Mercury’s LoadRunner
Compuware’s QALoad
Segue's Silk Performer
Empirix’s eLoad

Software testing support tools

•Defect tracking tool
Compuware’s TrackRecord
•Configuration management
MS Visual Source Safe

Summary

•Have a clear mission – Plan, Plan, Plan !!! (Define your What, When and How)
•Focus on the most important areas first
•Automation is more than building scripts
•Test Automation is an investment
•Treat test automation as a “development project”

Test Categories and objectives

Test Categories and objectives

Interactivity ( Pull down menus, buttons)
- Layout
- Readability
- Aesthetics
- Display characteristics
- Time sensitivity
- Personalization

•Using specialized Test Labs a rigorous testing process is conducted to get quantitative and qualitative data on the effectiveness of user interfaces
• Representative or actual users are asked to perform several key tasks under close observation, both by live observers and through video recording
•During and at the end of the session, users evaluate the product based on their experiences

Recovery Testing

•A system test that forces the software to fail in variety of ways , checks performed
- recovery is automatic ( performed by the system itself)
- reinitialization
- check pointing mechanisms
- data recovery
- restarts are evaluated for correctness
•This test confirms that the program recovers from expected or unexpected events Events can include shortage of disk space, unexpected loss of communication

Documentation Testing

•This testing is done to ensure the validity and usability of the documentation
•This includes user Manuals, Help Screens, Installation and Release Notes
•Purpose is to find out whether documentation matches the product and vice versa
•Well-tested manual helps to train users and support staff faster

Configuration test

•Attempts to uncover errors that are specific to a particular client or server environment.
•Create a cross reference matrix defining all probable operating systems, browsers, hardware platforms and communication protocols.
•Test to uncover errors associated with each possible configuration

Regression Testing

•Regression Testing is the testing of software after a modification has been made to ensure the reliability of each software release.
•Testing after changes have been made to ensure that changes did not introduce any new errors into the system.
•It applies to systems in production undergoing change as well as to systems under development
•Re-execution of some subset of test that have already been conducted
•Test suite contains
- Sample of tests that will exercise all software functions
- Tests that focus on software functions that are likely to be affected by the change
- Tests for software components that have been changed

User Acceptance Testing

•A test executed by the end user(s) in an environment simulating the operational environment to the greatest possible extent, that should demonstrate that the developed system meets the functional and quality requirements
•Not a responsibility of the Developing Organization

Acceptance Testing

•To test whether or not the right system has been created
•Usually carried out by the end user
•Two types are :
ALPHA TESTING :Generally in the presence of the developer at the developers site
BETA TESTING : Done at the customers site with no developer in site

Exploratory Testing

•Also known as “Random” testing or “Ad-hoc” testing
•Exploratory testing is simultaneous learning, test design, and test execution. (…James Bach)
•A methodical approach-style is desirable

Exploratory Testing - Tips

•Test design Crafting
•Careful Observation
•Critical thinking
•Diverse Ideas
•Pooling resources (knowledge, learnings)

What Is a Test Strategy?

•It provides a road map that describes the steps to be conducted as part of testing, when these steps are planned and then undertaken, and how much effort, time and resources will be required.
• It must incorporate test planning, test case design, test execution and resultant data collection and evaluation

Debugging

•Occurs as a consequence of successful testing
•Is an action that results in the removal of the error
•Results of the test give a “symptomatic” indication of the software problem

•The symptoms and the cause may be geographically remote.
•The symptoms may be caused by human errors.
•The symptom may be because of a timing problem rather than processing problem.
•The symptom may be intermittent

First step in fixing a broken program is getting it to fail in a repeatable manner. – T.Duff

•Testing is a structured process that identifies an error’s “symptoms”
•Debugging is a diagnostic process that identifies an error’s “cause”

Methods/Approach used for debugging :
• Brute force
• Cause elimination – Induction or deduction
• Backtracking

Example of debugging by Brute Force are
1.By studying Storage Dumps I.e. usually a crude display of storage location
2.by invoking run-time traces
3.by scattering print statements
4.by use of automated debugging tools

Cause elimination
Debugging by Induction
1.Locate data about what program did correctly/incorrectly
2.Organize data
3.Device a hypothesis about the cause of the error
4.Prove the hypothesis
Debugging by deduction
1.Enumerate the causes of error
2.Eliminate each cause of error

Debugging by Backtracking

Beginning at the place where the symptom is uncovered, the source code is traced backward until the site of the cause is found

Error Analysis - When Correcting the error , ask these three questions :
- Is the cause of the bug reproduced in another program?
- What “next bug” might be introduced by the fix that I’m about to make?
- What could we have done to prevent this bug in the first place?

Metrics

•Why Measure
Tracking Projects against plan
Take timely corrective actions
Getting early warnings
Basis for setting benchmarks
Basis for driving process improvements
Tracking process performance against business
objectives

Testing Metrics

Defect Density
•Total Defect density = (Total number of defects including both impact and non-impact, found in all the phases + Post delivery defects)/Size
Average Defect Age
•Average Defect age = (Sum of ((Defect detection phase number – defect injection phase number) * No of defects detected in the defect detection phase))/(Total Number of defects till date)
Defect Removal Efficiency
•DRE = 100 * No. of pre-delivery defects / Total No. of Defects

Review Effectiveness
•Review Effectiveness = 100 * Total no. of defects fond in review / Total no. of defects
Cost of finding a defect in review(CFDR)
•Cost of finding a defect in reviews = (Total efforts spent on reviews / No. of defects found in reviews)
Cost of finding a defect in testing(CFDT)
•Cost of finding a defect in testing = (Total efforts spent on testing / defects found in testing)

Cost of Quality
•% Cost of Quality = (Total efforts spent on Prevention + Total efforts spent on Appraisal + Total efforts spent on failure or rework)*100/(Total efforts spent on project)
•Failure cost = Efforts spent on fixing or reworking the pre-delivery defects + (3 * efforts spent on fixing or reworking the post-delivery defects)

Test Case Effectiveness
•Test Case Effectiveness = # of defects detected using the test cases * 100/ total # of defects detected in testing
•This metrics defines the effectives of the test cases which is measured in terms of the number of defects found in testing with using the test cases
•Source of Data
- Defect data and number of test cases from PMS
P.S.: - These metrics are mainly applicable to V&V projects

Test Case Adequacy
•Test Case Adequacy = No. of actual Test cases * 100 / No. of test cases estimated
•This metrics defines the number of actual test cases created v/s the estimated test cases at the end of the test case preparation phase
•The estimated No. of the test cases are based baseline figures and then added to ePMS
•Number of Actual Test cases is also derived from ePMS
P.S.: - These metrics are mainly applicable to
V&V projects

Test Case Adequacy
•Test Case Adequacy = No. of actual Test cases * 100 / No. of test cases estimated
•This metrics defines the number of actual test cases created v/s the estimated test cases at the end of the test case preparation phase
•The estimated No. of the test cases are based baseline figures and then added to ePMS
•Number of Actual Test cases is also derived from ePMS
P.S.: - These metrics are mainly applicable to
V&V projects

Defect Detection Index
• Defect Detection Index = # of defects detected in each phase / total # of defects planned to be detected in each phase
•This is a measure of actual v/s planned defects at the end of each phase
•Source
- Defect data from PMS
P.S.: - These metrics are mainly applicable to V&V projects

Principles of Testing

Principles of Testing

Economics of Testing
It is both the driving force and the limiting factor
Driving - Earlier the errors are discovered and removed in the lifecycle, lower the cost of their removal
Limiting - Testing must end when the economic returns cease to make it worth while i.e the costs of testing process significantly outweigh the returns
Exhaustive Testing

•Testing every possible input over every possible output
• Can use every possible input condition as a test case

Is Exhaustive Testing feasible?

E.g a COBOL Compiler
• Impossible to create test cases to represent all valid cases.
• Impossible to create test cases for all invalid COBOL Programs
•The compiler has to be tested to see that it does not do what it is not supposed to do
E.g to successfully compile a syntactically incorrect program
•Exhaustive testing is hence impossible
•Implications: One cannot test a program completely to guarantee that it is error free
Economics
•Objective is to therefore find maximum errors with a finite number of test cases

Limitations of Software Testing

Even if we could generate the input, run the tests, and evaluate the output, we would not detect all faults:
•Correctness is not checked:
The programmer may have misinterpreted the specs, the specs may have misinterpreted the requirements
•There is no way to find missing paths due to coding errors

Psychology of Testing

•Test Engineers pursue defects not people
•Don’t assume that no error(s) will be found
•Test for Valid and Expected as well as Invalid and Unexpected
•The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section
•Testing is extremely creative and intellectually challenging

Test Case

•“A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.”
(…IEEE)
In other words, a planned sequence of actions (with the objective of finding errors)


A Good Test-Case
•Has a high probability of detecting error(s)
•Test cases help us discover information (.. Kaner)
e.g. of information objectives
• Help managers make ship / no-ship decisions.
• Minimize technical support costs.
• Assess conformance to specification.
• Minimize safety-related lawsuit risk.
• Verify correctness of the product.


Other Terminologies

Test Suite – A set of individual test cases/scenarios that are executed as a package, in a particular sequence and to test a particular aspect.
E.g. Test Suite for a GUI or Test Suite for functionality
Test Cycle – A test cycle consists of a series of test suites which comprises a complete execution set from the initial setup to the the test environment through reporting and clean up.
E.g. Integration test cycle / regression test cycle