Regression Testing
• We
all know - Regression testing is a type of software testing that seeks to
uncover new software bugs, or regressions, in existing functional and non-functional
areas of a system after changes such as enhancements, patches or configuration
changes, have been made to them
• A
regression test can be a manual test or a automated test at different layer of product – unit
test, functional test, api test or a web service test
• Continuous
evolving of product with new features, enhancements and technology change
contribute to regression test suite over product releases
• Regression
testing is time consuming effort given the multiple code lines , service packs and hot fixes
What happens?
The Traditional Way
• Traditional
SDLC brings regression down in testing phase, where we execute all the regression test suite both manual and automated
• Uncover
of bugs late in the game
• Costly
in terms of time and resource
• Quite
difficult with multiple code line , Service Packs and HOT fixes
• Is
there a better way to do this?
During Development
• Developer
writes code (new feature , enhancement or bug fix) and executes the
corresponding unit test cases and submits code to source control system
• Nightly
build or continuous integration are run to check the code fix however at unit
level , again all the unit tests are executed
• Can
we do something here?
Agile World
• User
stories are addressed in sprints , test case authoring , automation and
execution happens sprint wise however execution may happen in cumulative
fashion
• Does
the above guarantee shippable product after each or few sprints?
• Hardening
sprint to address regression testing
• Can
it be still better?
Regression Test Analyzer - A way to address questions above
• A
tool which will find the most bug prone areas in your product by establishing a
relation between product source code and functional areas or tests , we can
implement this tool in two ways
– Through
Source control system(SVN or Perforce)
– Through
code coverage(test execution on instrumented code)
• The
rationale behind this tool is simple as we all know that changes to code make
product bug prone and targeting those areas for testing will be fruitful
• Bring
regression early in cycle and more importantly be SELECTIVE about the areas of
concentration for testing your product for regression testing
Through Source Control System
• Product
source code is organized in logical packages in a structured manner which helps
to build this tool, below are steps involved
– Simple
diff command to get the changes for a give date range , extract the change
information into a log file(XML format), you will get the change log file with package
– Process
the log using any programming language (java) to get the package(module or functional area) level information like name and number of occurrences of a
package and create a report
• For
example the report will look like this
Module
name Occurrences
mod1
20
mod2
12
mod3
5
Likewise list
of modules with occurrences will become input for our regression testing which
will help us to be selectively target and improve the product quality as it
gives insights to the more bug prone areas due to code changes
Below is the flow diagram
Through code coverage
• In
this approach we will establish a relation between product source code and
tests through code coverage
• We
instrument code to a get metadata file
• We
execute all the test to get the coverage file , we get a coverage file for executed
tests once we exit JVM
• To
get a code coverage report we use metadata and coverage file as input, which
gives the report at package level and class level
• To
get the report for individual test or set of tests, get the coverage file for
each test or set of tests by exiting JVM using code coverage tool API.
• This
way we can simply establish the relation between test and package level to product
source code
Conclusion
• Regression
Test Analyzer is a tool to analyze the areas for performing regression testing.
• The
output from tool can be used to cut the traditional regression cycles
• Bringing
regression up in the cycle to perform regression in development phase
selectively to uncover bugs
• This
tool helps in getting quick feedback as we use selective approach, this is
useful in agile world
• Saves
time and resources
• Improves
product quality
Possible Enhancements
- A eclipse plugin
- Integration of automated tests suite to kick off based on report
