Non Functional Testing

The Cycle

Each of the following stages are carried out by a Non Functional Test specialist:

Test Script Development Stage

Test Script development usually occurs against the same interfaces used by real users.  There are however exceptions to this when no direct user interface exists, for example scripts that pass web service requests might use *WSDL files to generate the scripts - short for Web Services Description Language, an XML-formatted language used to describe a Web service's capabilities as collections of communication endpoints capable of exchanging messages.

During this stage the Non Functional testers generate test scripts using an API capture tool such as LoadRunner’s Virtual User Generator or Silk Performer's script recorder.  As only one or two user ids are required for this process scripting can occur on an environment that is not scaled for Non Functional test execution.  Scripts are developed with parameters to pass through multiple sets of user data such as User Ids and Passwords. The scripts are coded to ensure that errors are handled and reported.  Scripts are developed to handle system and data related errors.  Transaction timing points are coded into the test scripts to measure response times for individual server requests.  Scripts are also coded with user think times to ensure that the script, when executed, can simulate the pauses between server requests as incurred when a real user executes the business transaction.  Scripts are also coded so they can reiterate.  During the test script development stage any test data is identified that allows for multiple users executing the scripted business transactions within a Non Functional scenario.  This data may need to be sourced from the application database via queries, generated externally, derived from system responses or randomly generated.

Test Scenario Development Stage

During this stage the test scripts are grouped into test scenarios. The scenarios allow for configuration of user load profiles.  The scenarios can be configured to simulate the user activity as defined by the volumetric analysis. A series of scenarios can be developed to reflect a range of user activities to test the system stability.

System Monitoring Stage

In order to correlate user activity with system stability and to identify points of system failure, the system servers are required to be monitored.  Where possible the Non Functional test tool is configured to monitor the servers.  Monitored statistics are taken from the server’s native resource monitoring utilities and then imported into the test tool for reporting after the test has concluded.  The system monitoring generally requires user ids and passwords, with sufficient access rights, to be provided to the Non Functional test team.   This requirement will be identified in the Non Functional test strategy as a deliverable from the project group to the Non Functional testers.

Test Execution Stage

The test execution, test analysis and issue resolution phases form an iterative process. A period of testing is followed by a period of analysis then an issued resolution followed by more testing.  This approach is aimed at identifying and driving out the major issues first, then on to the next issue.  Some applications perform perfectly first time and do not require this iterative process; some - but not many!  During this stage a detailed test diary is kept to record all test execution activity.  The objective is to identify and, where possible, drive fixes for Non Functional bottlenecks.

During test execution an open conference call is established by the Non Functional testers.  Key project technical resource are encouraged to participate in the analysis of the system during test execution, indeed this may be a requirement.  This helps expedite the issue identification and resolution process. These people might include:

•    DBA’s

•    Network administrators

•    Developers

•    Server administrators

Test Reporting Stage

During the iterative test execution and tuning stages reports are generated and sent to the project group.  These reports are ad-hoc and usually in the form of emails.  This allows for rapid cycles of testing and tuning whilst updating the project group on progress and issues.

Final Test Report Stage

At a point where test execution demonstrates a stable system, or a system that is stable as possible within the Non Functional Testing window, a final report is generated based on the results of the final test execution. The Final Test report generally includes the following:

•    Management summary providing high level overviews and key findings, without too much technical analysis

•    Detailed technical results of the Non Functional Testing and tuning effort

•    Detailed metrics against the required Service Level Agreements and performance requirements as detailed in the initial test strategy

•    Issues identified and rectified

•    Recommendations

•    Conclusions

The final report measures the results and testing activities against the approved Non Functional Test Strategy.  Any changes to the scope of testing agreed during the testing stage will be detailed in the Final Test Report.  The Final Test Report measures the results against the initial Non Functional Test Strategy ensuring that the test results are measurable and auditable.