mBrace Modelling - Data Gathering

Test Automation or Scripting for Performance Modelling

Loadrunner

Before LoadRunner measurement collection can be made, any time differences between application servers and client PCs must be established, including the load generator PCs.

If possible, the time settings on the load generator PCs, in fact all machines and servers should be synchronised. At the very least, the difference in time between the various servers should be recorded. During the measurement collection, measurement scenarios are performed.

A measurement scenario may consisit of one or more measurement scripts. A measurement script consists of transactions which need to be executed during the measurement session.

The transactions are inserted into the script in the order prescribed by the business process. Each script is directly related to a business process or use case.

 

Single User Measurements

The measurement scenario consists of a range of use case related scripts. There is a script for each use case. The transaction types of the use case are executed one at a time and the use case is then repeated in five additional iterations, for a total of six iterations. The transaction types contained within each script are executed with 5 second fixed user think time intervals. The fixed length user think time intervals between iterations is 10 seconds. No random user think times should be used. The transactions contained within the script are executed using a single user. Stacked single user measurements can be executed using a number of different Vusers.
The Loadrunner scenarios should be tested and checked for proper operation prior to implementation. An exclusively available test environment is not required for script verification. The Loadrunner scripts should be made available to the measurement engineer at least two days prior to the measurement session. It is possible to make an exception, if for example Loadrunner can only function correctly if each script can only contain a single transaction type. In such a case, the single transaction scripts are executed in the order determined by the use cases. The original script therefore becomes a group of one-transaction scripts.
Immediately following the single user measurement, the Loadrunner engineer should perform the following actions:
• Check the correct duration of the measurement session and report these details via email to the performance analyst
• With the help of LoadRunner Analyzer, create a standard report in Word format
• Gather a set of raw data (the LoadRunner log) and deliver this to the performance analyst (via email or on a shared drive) 
• The Standard report should contain the content of the example report as shown in Appendix 1 of the manual 
For details on the log format, please see Data Gathering - Submission.

 

Multi User Measurements

A pilot measurement exercise is usually conducted prior to the actual multi user measurement session. This is preferably done at least one day prior to the actual measurement session. On the basis of the pilot measurement session, it is possible to determine the limitations of transaction volumes on the test environment before the test infrastructure becomes overburdened due to capacity limitations. The multi user measurement scenario is conducted using a “stepped” approach where a number of Vusers are used to execute scripts for 10 minutes before increasing the number of users to the next step. The stepped increase in vusers occurs every 10 minutes until the maximum user target is reached, as was established in the pilot measurement session.
Transaction user think times are identical to those encountered/used during the Single User Measurement session. Following the completion of the Multi-User scenario, a standard Loadrunner report should be created using the Loadrunner Analyser utility and the Loadrunner raw data logs should be delivered to the performance analyst, in a manner similar to that encountered in the single user measurement scenarios described above.
The Loadrunner sections which include Running Vusers, Transaction Rate and Transaction Summary are of particular importance.

 

Logging in Loadrunner

This section lists a number of files which are of interest to the mBrace Performance Analyst.


The example template file “20070827_mBrace_Loadrunner_Template.usr”  will be used for demonstration purposes.


The “20070827_mBrace_Loadrunner_Template.usr” file contains a number of actions that can be reused. Details of these script actions are briefly explained below:


vuser_init


The vuser_init action contains declarations of the necessary variables, the VTS dll "vtclient.dll”  is loaded and the connections are made with the two VTS instances. One VTS instance is used to co-ordinate the Stacked Single User measurements (port 8888) and the other instance is used for logging purposes (port 9999). If the port numbers mentioned above are unavailable, other ports may be used.


globals.h


Includes the VTS header file:


#include "vts2.h"


Test_Init


Contains a number of required test parameters. These parameters are provided by the Controller to the load generators executing the scripts and include the following:


"Application": Name of application;

"MeasurementVersion": Version number or name of measurement, for eg. “20070807_Prod”

"MeasurementType": SU, BO of MU

" MeasurementTypeVersion ": Version of the measurement type, for eg. “BO-02”


The values can be passed from the Controller using command line parameters:

-    Select the relevant group in the Controller,

-    select Details

-    select More

-    Fill in the following example command line parameters:


“-Application ZBP  -MeasurementVersion 20070807_Prod -MeasurementType BO -MeasurementTypeVersion BO-02”


On the basis of the provided MeasurementType information, the value of a number of variables are determined. The value of the MeasurementInterval variable determines the wait time between the sub-measurements of a stacked single user measurement. This should only be specified if the application encounters difficulties with stacked single user scenarios that do not have wait times between stacked transactions. The TT value is the think time used for the various types of tests used. The measurement engineer should enter the correct think time value for the test.

NB: If the script is run from VuGen, the lr_get_attrib_string will not work correctly, and variables can be entered instead using the lr_save_string functions.

NB: This action occurs after vuser_init in the Init section. In addition, ensure that the “Initialize all vuser before starting” option has been selected on the Controller.


VTS


The VTS action contains three functions. Each function is described below:


VTS_Init()  -This function checks whether the VTS server is ready at the start of the script, and checks whether a token is available.


VTS_Start()   -This function is used to check for the availability of a VTS token. If so, the token is removed and two time stamps are recorded, one period time stamp and one readable time stamp. The user then continues with the transaction.


VTS_Gereed(char* Transaction)  -This function collects all relevant data during the measurement session, formats the data to a specified format and makes an entry to the VTS server.


The functions VTS_Start() and VTS_Gereed(char* Transaction) are called from the relevant transaction script. In the template, the action function is called Usecase.


Usecase


In order to service the scripted transactions, a number of functions are called. First, the usual lr_start_transaction and the lr_end_transaction. The transaction name is composed of 3 components:


<MeasurementId>;{Iteration};<MeasurementDescription>, whereas {Iteration} is a Loadrunner parameter of type iteration number.


The lr_start_transaction and the lr_end_transaction functions are surrounded by VTS_Start() and VTS_Gereed(lr_eval_string("<MeasurementId>;{Iteration};<MeasurementDescription>")).


With the rule


Response time = lr_get_transaction_duration(lr_eval_string("<MeasurementId>; {Iteration};<MeasurementDescription>"));


the transaction response time is written to a variable which is used for logging purposes.


After every VTS_Gereed function call, there is an lr_rendezvous("<rendezvouspointName>") function call. Note that each rendezvous point requires a unique name.


Note: If a multi-user measurement is being performed, rendezvous points in the script should be disabled at the Controller via Scenario – Rendezvouspoints!

 

vuser_end


In this action, the connections to the two VTS instances are broken (See known issues below)


Using VTS


Before a measurement scenario is executed, both VTS instances should be started. The instances can be started through the VTS GUI (vtconsole.exe) on the Controller machine, or another machine. One instance serves to co-ordinate the Stacked Single User (BO) measurements (port 8888) and the other instance serves to provide a metrics logging capability (port 9999). Enter the respective port number in the “VTS Port Address” option box and select “Start”. The correct column names can subsequently be imported via Options –Import data option. The existing VTS_BO.txt  file is used for the BO co-ordination instance, and the VTS_Logging.txt file is used for the VTS instance dedicated to the logging requirement.


When a measurement has been completed, the VTS results can be exported to a “.csv” file using the Options – Export data functionality.

 

Known Issues


•    At the end of a BO measurement session a runtime error appears if the vusers attempt to terminate their connections to the VTS server simultaneously. The root cause of this runtime error is still under investigation. The runtime error does not appear to be a problem, as the VTS GUI just keeps running and the log data can be exported without any issues.

•    If one of the sub-measurements of a BO measurement encounters a fault during execution, the VTS co-ordination mechanism no longer functions correctly as the token is not returned to VTS server instance for the next Vuser. This can be prevented by identifying possible failure points in the script, capturing fault events, and returning the token to the VTS instance using the VTS_Gereed function. In case of such failure events with a BO measurement, the analyst should be wondering what the value of a BO measurement is if it contains failed transactions.


Measurement data should be “zipped” up and delivered to the mBrace Performance Analyst as soon as they become available. Any measurement data should not be deleted, unless the mBrace Performance Analyst has indicated that the measurements were safely received and that it is safe to delete the data. It would be prudent to ensure that at least two copies of the measurement data exist at any one time.