Loadrunner Standards - A Useful Guide- Performance Testing Techniques

Introduction

So, you’re a test manager, responsible for some Loadrunner performance testing, but you don’t know anything about it. Help is at hand, here go some standards which if you like, you can insist on. When scripts are completed, ask to have a look at them, with these guidelines, you should be ale to have a good stab at ensuring decent standards are adhered to.

Script Structure

A virtual user script can be made up of a number of different sections, each performing a different action, with each section handled differently by the compiler and controller during test execution.

- vuser_init: The initialisation phase of the script

- action: The body of the business process, to be iterated

- more actions: Further actions within the process

- vuser_end: The close down, or housekeeping phase of the script.

When recording the business process and enhancing the virtual user script, the following standards must be applied to the structure of the script to ensure it is recorded in a consistent and supportable manner.

Vuser_Init

This section is executed once, when the virtual user is brought up and initialised, and contains any information necessary to enable the virtual user to log onto an application. It will also hold any directives to load function libraries, and to hold any functions unique to that particular script. Often, it will contain nothing but the script header block.

Generally it will contain:

  • - File references
  • - Define directives
  • - Script headers
  • - User defined funtions
  • - User logon details

Each virtual user script contains exactly one ‘init’ section, and therefore there is no need for it to be renamed.

Action

The action section is the part of the script that performs the iterative work of the business process. The contents may be iterated hundreds if not thousands of times, and therefore should not hold any irrelevant content such as functions.

The default name is 'action'.  Even in a single action script, the main transaction should be named.  Thus, if the script is run with other scripts in a mixed traffic test, each type of action may be clearly distinguished in the resultant outputs.

In a more process intensive script, more than one action may be involved to enable tighter control on processing and flow control. In this case, each action should be given a meaningful name within the script, which need follow no standard convention save that it needs to be readable. Generally, if a script can be broken down into logical units, then the use of individual named actions should be seriously considered.

Vuser_End

As with the ‘init’ section, this section is executed only once, either after the completion of the appropriate number of iterations, or by initiation of the controller during scenario ramp-down.

Should it be appropriate for the application and business process to log off in a controlled manner, it will occur in this section. It serves no other purpose, and occurs exactly once in each script. There is no requirement to rename this section.

Error Checking

When running test scripts either as part of the baseload or when specifically testing an application, it is important to ensure that the script is performing correctly and simulating a real user as closely as possible. It is possible for a script to fail and no longer be sending or receiving valid data to the application, but without proper error checking the script would carry on regardless.

Error checking ensures that in the event of a problem, the script stops running and reports the error back to the operator. This will also prevent the tests being run when the responses from the application are invalid.

Run-time parameters:  Avoid the use of “continue on error”

HTML vs HTTP

When recording web based scripts, there are 2 different modes HTTP (URL) and HTML (from LoadRunner version 7.0 referred to as "URL based"). These methods differ in the way they record the script.

  • - HTML is based specifically on user actions and the content returned by the application in response to those actions.
  • - HTTP is based on the requests generated by the client and sent to the server, such as a web page, or a graphic.

    When recording Web based scripts, HTML mode shall be used wherever possible, with HTTP mode being used only if specified during the Application Analysis or if HTML cannot be made to work (eg recent FOODS scripts).

Security

There are two methods of ensuring security that need to be accounted for during scripting - SSL or Secure Sockets, and Digital Certificates. It is important to note that once scripting has commenced, should either method be applied or changed then the scripts must be re-recorded to take into account the changes.

It is not possible to change the URL from http: to https: by simply editing the script, therefore to prevent rework the extent of SSL security within an application under test must be defined within the Application Analysis.

The extent of Digital Certificate use within an application is to be defined within the Application Analysis. Any certificates required must be provided and implemented in both the scripting machines and the Load Generators before scripting commences.

Transactions

The various transaction points encountered in each test script must be defined for ease of management and understanding of other personnel in the testing team. These transactions must be defined in the Application Analysis.

To ensure that all transactions are compiled in a similar manner, the following standards must be adhered to:

- Where possible, transactions are to be created during script recording.

- Transactions are to be named using a number unique to that particular script, and an descriptor indicating the function of the transaction:

P01-<Descriptive Transaction Name>

- Transactions are to be placed where an activity will initiate interaction with the server.

- Transactions are to be started and ended immediately before and after each interaction is completed; other elements, such as correlation and flow checking are not to be included in the transaction.

- Transactions are not to encompass think times, with the exception of the overall transaction, which will be identified as:

P01-<ScriptName>_Overall

- Deviation to this is allowed, but if transactions do include think times then it is to be detailed in the test plan.

- Under normal test conditions, do not nest transactions. If this is required, for example in an overall transaction, it should be documented within the Application Analysis.

Think Times

Using think times within a LoadRunner script is the method for providing emulation of a realistic business process, running at a realistic speed.

During script recording, the recording setting must include think times as recorded. If for whatever reason, the think times that are recorded are not realistic, then the times scripted must be adjusted to reflect testing objectives.

If transactions are inserted during recording, be aware that this will distorting the think times. In this case, manual correction will be required.

During execution, to introduce the random element that would be present in any business environment, standard runtime settings are to be set for between 80% and 120% of recorded think time. This will ensure a 20% deviation, and ensure a complete mix of business processes during execution. This runtime setting can be amended if required by the project, but this amendment must be detailed in the test plan.

Rendezvous

Rendezvous points are inserted into a script to cause a virtual user to pause at a known point in the execution of a script and to then wait for a percentage of the running users to achieve this state before continuing. This simulates the situation where large numbers of users are competing for the same resources simultaneously.

This causes a lot of strain on an application, and should not be used unless specified in the Application Analysis or Test Plan. If the rendezvous point is to be placed in a script that will take part in a mixed scenario, it should be used in a copy of the script that should be documented and saved separately, removing the possibility of the inadvertent use of the rendezvous and therefore invalidating the test run.


Comments

A virtual user script can be complex and difficult to understand, especially the application processes being employed. To ensure that test scripts are manageable and comprehensible to other members of the test team, comments should be used wherever possible to demonstrate the scripts purpose, transaction usage and process logic. The following standards apply to the use of comments within a test script:

- A commented header is to appear that the beginning of each script in the 'vuser_init' section.

- The header must record the following information:

- VUser script filename;

- Application name or code;

- Author;

- Creation date;

- Version information & Change history;

- Brief Description;

- Data dependencies.

- The header must be amended each time the script is adjusted or re-recorded.

- Place comments at the arrival point of each loaded page or results page in the process navigation.

- Place comments at any points where script actions may need clarification during isolated testing, or for other reference purposes.

- Where cookie references are retained in a script, their use must be commented to provide information about the cookie.

Note:  multi-line comments begin with '/*' and end with '*/', while single line comments begin with '//'. Either may be used at the discretion of the script developer.

Include Files and Common Function Libraries

The use of function libraries is recommended to provide extra functionality to the standard LoadRunner functions during the execution of a test. In order to ensure the effectiveness and successful management of function libraries, a series of standards are to be followed.

Functions that are created to provide generic functionality to a script such as file output or text processing are to be placed in an external function library and made available throughout the test rig. This file is to be named and placed according to the naming conventions outlined below.

Functions that are designed to overcome an issue within a single script, but are of little or no value outside that particular business process are to be implemented within the 'vuser_init' section of the script itself, and its use and functionality documented in the header block.

Occasionally the Load Generator will not pick up an external function library from its network location. To remedy this, insert a line similar to the following into the scripts .usr file. This ensures that the relative path to the library files is passed to the Load Generator.

[ExtraFiles]

..\..\data\lib1.h=

..\..\data\lib_abc.h=

For more information on function libraries, see the LoadRunner user guides.

Cookies

Cookies are widespread in web-based applications. Cookies are used to record information on the client for use by the server, or to maintain state during use of an application, and are generally unique to a particular user. This can create a problem when executing multiple virtual users on the same load generator, and so the following standards must be adhered to:

- If cookies are required for an application to function correctly, their use should be documented in the Application Analysis.

- The use of cookies within a script should be correlated to allow the script to run as multiple virtual users. Note that LoadRunner normally handles cookies dynamically and only if they are observed to fail should they need to be correlated.

- When a cookie is managed within a script, comments are to be used to provide information about the cookies function, and the methods used to control it.


Script Flow Checking

Flow checking functions are to be used within a script to ensure that the recorded business process is executed according to specification. In the event that an error occurs or an incorrect page is loaded, the checks will perform actions to continue the test automatically after flagging the error for later analysis.

There are three methods of flow checking in an application, not all of which are available for a particular protocol:

1. Screen ID's (including bitmap checks)

2. Other Parameters

3. Text Checks

Screen ID's.

Screen ID's (either visible or invisible) are to be used whenever available. In the case of graphical protocols such as Citrix, flow is to be confirmed using bitmap checks of part of the screen.

Other Parameters.

If a value is expected to be returned by the server and this value is required for further processing, then the collection of the parameter using web_reg_save_param can fulfil the flow control function. This must be commented into the script.

Text Checks

If there is no other indication of the correct page, then text checks can be used. This is usually the simplest and most successful method of page checking, and so may be used if desired. All checks are to be commented when used. To perform these checks web_find should be used, although it is important that the implications of this function are understood.

If inserting web-find during recording, be aware that the recording places the text before the called URL, which may place it WITHIN a transaction.  In line with xxxxx above, the web-find command should be moved outside the transaction wherever possible (but still before the call).

Status & Error Messages

Messages are to be used within a script to relay status or error messages to either the controller console window, the execution log, or both. Their use is to be judged on a script-by-script basis, although it must be borne in mind that each message has a network overhead that may skew results if over-used.

There are two distinct types of messages that are used in LoadRunner scripts:

- Status messages, including Iteration Status

- Error messages

An Iteration Status messages is used to relay information about the current execution state of a script, such as the number of iteration achieved, or that a particular point has been arrived at. These messages are to be constructed using the lr_vuser_status_message (message only to the controller window) function.

Think time and rendezvous messages should not be used unless specified in the Application Analysis. The only recommended message is the iteration count message, using the iteration number parameter:

lr_vuser_status_message ( "Iteration #%s", lr_eval_string ( "{iteration}" ) );

Error messages are to be framed using the lr_error_message (message only to the output window) or lr_output_message (message to both the output window and the Vuser log) functions. All messages are to commence with the word 'ERROR' in upper case. It should then briefly describe the error type and give any meaningful information to highlight the error as it appears in the logs.

More information concerning messages can be found in the LoadRunner manuals.

Parameterisation & Correlation

The need for parameterisation and correlation within a test script is an important factor in ensuring the script can be run with multiple users when it comes to test execution.

Parameterisation is important since it can be used to vary the data used in requests sent to the application. This simulates activities that are closer to real user interaction, and consequently simulate the appropriate load on the application infrastructure. Often the only way to use multiple users is to parameterise data, for example the username and password in an application configured to deny multiple logins.

Correlation is used to use data that has been received by the client in the framing of appropriate requests subsequently sent to the server. A common use of correlation is to create parameters from session ID data.

Parameterisation and correlation are to be used whenever necessary.  Usage must be documented in the Application Analysis, and comments must be added to the script.

Parameter names are at the discretion of the developer, but must be meaningful. As a guide, parameters that perform the same function within multiple scripts, such as a parameter to cover user ID's, should be given the same name in all scripts.

All parameters are to be stored in comma delimited text files within the data directory on the shared infrastructure (see section XXX). When selecting a parameter from a data file, the column name (which is actually the parameter name as well) must be selected, rather than the column number.

Logging

For early tests with low workloads, a high level of logging can help to debug the load test. Later on in the testing process when the test is more robust, the level of logging can be reduced or switched off.

Logs should always be kept, even for tests which are not deemed to be successful as they can be useful in determining the cause of a performance problem.

Naming Conventions

Naming conventions should be devised for:

- Transactions

- Scripts

- Scenarios

- Results

- Analysis sessions