Software Regression Test Analyzer

Regression Testing 

•       We all know - Regression testing is a type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes such as enhancements, patches or configuration changes, have been made to them

•       A regression test can be a manual test or a automated test at different layer of product – unit test, functional test, api test or a web service test

•       Continuous evolving of product with new features, enhancements and technology change contribute to regression test suite over product releases

•       Regression testing is time consuming effort given the multiple code lines , service packs and hot fixes

What happens? 

The Traditional Way

•       Traditional SDLC brings regression down in testing phase, where we execute all the regression test suite both manual and automated  

•       Uncover of bugs late in the game

•       Costly in terms of time and resource

•       Quite difficult with multiple code line , Service Packs and HOT fixes

•       Is there a better way to do this?

During Development

•       Developer writes code (new feature , enhancement or bug fix) and executes the corresponding unit test cases and submits code to source control system

•       Nightly build or continuous integration are run to check the code fix however at unit level , again all the unit tests are executed

•       Can we do something here?

Agile World

•       User stories are addressed in sprints , test case authoring , automation and execution happens sprint wise however execution may happen in cumulative fashion

•       Does the above guarantee shippable product after each or few sprints?

•       Hardening sprint to address regression testing

•       Can it be still better? 

Regression Test Analyzer - A way to address questions above

•       A tool which will find the most bug prone areas in your product by establishing a relation between product source code and functional areas or tests , we can implement this tool in two ways

–      Through Source control system(SVN or Perforce)

–      Through code coverage(test execution on instrumented code) 

•       The rationale behind this tool is simple as we all know that changes to code make product bug prone and targeting those areas for testing will be fruitful

•       Bring regression early in cycle and more importantly be SELECTIVE about the areas of concentration for testing your product for regression testing

Through Source Control System

•       Product source code is organized in logical packages in a structured manner which helps to build this tool, below are steps involved

–      Simple diff command to get the changes for a give date range , extract the change information into a log file(XML format), you will get the change log file with package

–      Process the log using any programming language (java) to get the package(module or functional area) level information like name and number of occurrences of a package  and create a report

•       For example the report will look like this

                                   Module name                Occurrences

                                        mod1                                 20

                                        mod2                                 12

                                        mod3                                  5

       Likewise list of modules with occurrences will become input for our regression testing which      will help us to be selectively target and improve the product quality as it gives insights to the more bug prone areas due to code changes

Below is the flow diagram

 

Through code coverage

•       In this approach we will establish a relation between product source code and tests through code coverage

•       We instrument code to a get metadata file

•       We execute all the test to get the coverage file , we get a coverage file for executed tests once we exit JVM

•       To get a code coverage report we use metadata and coverage file as input, which gives the report at package level and class level

•       To get the report for individual test or set of tests, get the coverage file for each test or set of tests by exiting JVM using code coverage tool API.

•       This way we can simply establish the relation between test and package level to product source code

Conclusion

•       Regression Test Analyzer is a tool to analyze the areas for performing regression testing.

•       The output from tool can be used to cut the traditional regression cycles

•       Bringing regression up in the cycle to perform regression in development phase selectively to uncover bugs

•       This tool helps in getting quick feedback as we use selective approach, this is useful in agile world

•       Saves time and resources

•       Improves product quality   

Possible Enhancements

•  A eclipse plugin
• Integration of automated tests suite to kick off based on report