Test Automation & Tools

Test Automation & Tools

•What is test automation
•Why Automate
•Test Cases suitable for automation
•Test Cases unsuitable for automation
•Automation Strategies
•Pros & Cons - Manual Vs. Automated testing
•Let us get to know some Automation Tools
•Summary

What is Test Automation?

Replace manual Test Engineers by automating everything

Automated testing by itself will ensure high quality

Challenges of Testing

•Today we witness huge and very complex code churned out at faster pace
•Testing is struggling to keep pace with Technological obsolescence and Advancements
•Fast Changing requirements
•Waterfall Model of development leaves restrained scope for optimal testing
•Testing is often squeezed to the fag-end of the SDLC

How to overcome the challenges?

•Trade coverage for risk
Reduces scope
Shortens test cycles
•Outsource Testing work
•Introduce/ Enhance AUTOMATION

Why Automate?

•Repeatability : Tests can be executed multiple times and consistently each time.
• Leverage : Executing tests that were never performed manually at all.
• Accumulation : The number of test required for coverage increases with each enhancement done to applications

Benefits of Automation

•Improves human productivity
Broader test coverage
•Reduces redundancy
Automate repetitive tasks
•Enables process improvements
Increase regression testing
Expose risk areas
Better management of testing (with Test-Director etc.)

Automation MYTHS

•Automate everything
•Shorten the test cycle by 50%
•Automation is ‘Easy’
•An automated test is faster, because it needs no human intervention
•Automation will lead to "significant labor cost savings"

Suitable candidates for Automation

þTests that need to be run for every build of the application (sanity check, regression test)
þTests that use multiple data values for the same actions (data-driven tests)
þTests that require detailed information from application internals (e.g., SQL, GUI attributes)
þStress/load testing

þTest cases that can not be done manually
þScenario/ Use-Case / Transaction Flow test
þConfiguration test
þMulti-user Scenarios

Examples :

•While testing the MS word we can check if we open a new document 1000 times, default name should be “Doc1000”
•In the case of adding more number of data say for example 1lakh of data to the database
•When broken links or Orphanage links are to be identified from the web application
•While testing the MS word we can make different styles by using automation and can check

•After preparing the next build we can automate to test that build contains all the files and folders
•Performance testing of the server, network application at various anticipated load levels (if 100 users are logging to yahoo mail, we can find out the response time and behavior of the site)
•GUI properties like Size, Position, Label, Visible, Enable, Value etc. to be tested.(Customer requirement is that the company logo should be displayed at the top center in all the screens)

Unsuitable to Automate

ýAesthetics (ease of use, usability, look and feel)
ý One-time testing
ý Start testing immediately
ý Ad-hoc/random or Exploratory testing
ý Tests without predictable results
ý Testing dynamic areas under development

Why Automation Fails?

•Not clear or enough Requirements
•Staff skills may not match needs
•Staff turnover
•Insufficient time

Strategies for Automation

•Don’t try to automate everything right away
•Focus on the most important test cases first
Regression tests
High priority features or product requirements
High risk use cases, capabilities
•Start with repetitive tasks susceptible to human error if done manually

Manual Testing Pros & Cons

pros
•Quick & inexpensive
•Simulates real-world use better
•Better for extensive analysis of test cases as well as results
•Best when test steps and results are not well defined
•Can explore new test cases
•Does not require technically trained staff

cons
•Might not be consistent in re-running test cases.
•Re-running large volumes of test cases is expensive and tedious.
•Work-force turnover may upset plans

Automated Testing Pros & Cons

pros

•Can execute test cases unattended.
•Can cost-effectively run large volumes of test cases.
•Can cost-effectively re-run large volumes of test cases repetitively

cons

•Time-consuming & expensive
•Maintaining (modifications etc.) of test cases / scripts are difficult and expensive
•Requires a highly trained technical staff
•Re-running old tests may not necessarily spot new bugs

What Is a Test Script?

A series of commands or events stored in a script language file that execute a test case and report the results.

Fundamentals of Test Automation

•Maintainability
Applications are maintained continuously
Cross-reference test scripts o the application
Design to avoid regression
Adopt a test framework
•Optimization
Enough tests to do the job without having to manage to many

•Independence ( test cases)
Independent data
Independent context
Result independence
•Modularity (test scripts)
Identify common scripts

Synchronization
• Execute the test scrip and the application at the same pace
• Different conditions may exist at the time of playback than existed when the test was created, precise timing coincidence may not be possible

Test Framework

•Test framework is like an application architecture
•It outlines the overall structure for the automated test environment, defines common functions, standard tests, provides templates for test structure, naming convention
•Leading to a maintainable and transferable test library

•Common Functions (e.g. Setup, Sign on, Monitor, Log Error, Recover, Signoff, Cleanup)
•Standard Tests (e.g. Walk thru to navigate thru each menu item)
•Test Script templates – (Header : Test description, inputs, outputs, special requirement, Dependencies, Common routines like setup etc.,End))
•Test Dictionary –( Vocabulary, Naming Conventions, Cross reference names to application)

Test Automation Approach

Approach / Profile


Capture /Playback Application already in test phase or maintenance
Stable application.
Variable Capture/ Variable Playback
Application in planning, analysis or design
Unstable or stable application
Technical test team

Different type of tools

•Reviews and Inspection
•Test planning
•Test design & Development
•Test execution & verification
•Test support

Automation tools are not one-size-fits-all

Tools for Reviews & Inspection

•Requirement analysis
Compuware’s Reconcile/TeleLogic’s DOORS/Rational Requisite Pro
•Complexity analysis
Compuware’s Numega
•Code comprehension
Compuware’s Numega
•Syntax analysis
Lint for C and Jlint for Java

Tools for test planning

•Templates for test plan documentations
•Test schedule and staffing estimates
•Complexity analyzer

Tools for Test Design & Development

•Test data generator
Compuware’s File-Aid
•Requirements-based test design tool
•Capture/Playback

Test Execution and Evaluation Tools

•Capture/playback
Mercury’s WinRunner
Compuware’s QARun
Segue’s Silk Test
Rational’s Visual Test
Rational’s ROBOT
•Coverage analysis
•Memory testing

•Test case management
Mercury’s Test Director
Compuware’s QADirector
•Simulators & performance
Mercury’s LoadRunner
Compuware’s QALoad
Segue's Silk Performer
Empirix’s eLoad

Software testing support tools

•Defect tracking tool
Compuware’s TrackRecord
•Configuration management
MS Visual Source Safe

Summary

•Have a clear mission – Plan, Plan, Plan !!! (Define your What, When and How)
•Focus on the most important areas first
•Automation is more than building scripts
•Test Automation is an investment
•Treat test automation as a “development project”