Types of Performance Testing

Performance Testing – An Introduction

The term Performance can mean measuring response time, throughput, resource utilization, or some other system characteristic ( or group of them), by varying the number of users.

Performance Testing answer the below questions -

• Does the application respond quickly enough for the intended users?
• Will the application handle expected user load and beyond?
• Will the application handle number of transactions required by the business?
• Is the application stable under expected and unexpected user loads?
• Are you sure that users will have a positive experience on go-live day?

Performance Testing Process Image

Types of Performance Testing

Load Testing: Load Testing is done in order to check when the application fails by increasing the number of users and keeping the system resources as constant.

Stress Testing: Stress Testing is done in order to check when the application fails by reducing the system resources such as RAM, HDD etc. and keeping the number of users as constant. This is done to find out the breakpoint of an application.

Endurance (Soak) Testing: Endurance Testing is determining the system’s sustainability on a specified user load for long duration of time (may be for number of days).

Volume Testing: Testing where the system is subjected to large volumes of data. Testing which confirms that any values that may become large over time (such as accumulated counts, logs, and data files), can be accommodated by the program and will not cause the program to stop working or degrade its operation in any manner.

Important Terminologies used in Performance Testing

Important Terminologies used in Performance Testing :

Non Functional requirements (NFRs) : NFRs are the business requirements cater to Security Testing, Resilience Testing, Compatibility Testing, Recovery Testing, Usability Testing, Localization Testing, Performance Testing, Scalability Testing.

Performance NFRs are defined in one or more parameters - Transaction response time (min/max/average/90th percentile), hits per second, throughput, transactions per second under desire load. One or more of these parameters are also known as Key Performance Indicators (KPIs).



NFRs are also known as Service Level Agreement (SLA)

For Example :

Expected maximum user load - 10,000 users

Target concurrency of 2500 users (SLA)

Response time threshold (KPI) of 15 seconds

Transaction : Transaction is an end-to-end measurement that defines the start and end point of a user request which is responded by a server. For Example – “Clicking a Submit Button”, “Clicking on web link”



Transaction Response Time: Time take by a transaction to get responded or serviced by the server.



Concurrency: The group of users within a business processing environment that are requesting same processing services (functionality) from the environment all at the same point in time.



Simultaneous Users: The group of users within a business processing environment that are requesting different processing services (functionalities) from the environment all at the same point in time.

Think Time: The time between the browser displaying a page to the user and the user clicking a link to browse to the next page. This time could be the time it takes the user to read the content of the page or decide what to do next. Load testing tool records this time when recording a Business Case and uses it to accurately simulate the users when performing a test




Virtual User: A software entity generated by the performance testing tool, that simulates a real user by repeatedly performing a Business Case during a load test



Throughput: Number of Request Completions Per Unit Time or

Number of response units (bytes/kilo bytes) received from server per unit of time (seconds/minutes/hour)
Examples of throughput in Performance Testing:


Number of banking transactions per minute

Number of orders per minute

Number of web interactions per second

Number of disk I/Os per second, bandwidth


Case Study on Throughput:

At ABC Bank there are


2 million withdrawals per day,

2 million inquiries per day, and

1 million deposits per day.

75% of the transactions occur in a 3 hour window –11am to 2pm.

What is the peak throughput?

Solution :



Daily throughput = 2+2+1 = 5 million transactions/day Peak throughput = 75% of (5 million transactions per 3 hours)

= 0.75 x 5 million / 3 = 1.25 million / hour

VUGen Protocol Analyzer - How to analyse the protocol for recording in Loadrunner

Analyzing protocol for VUGEN recording :

Most often, it is found that tester attempts to record a web application but face some of the obvious challenges. What is the good deal then? Let me explain some of the situation in this context -

Query 1 : Hi, I am new to Loadrunner and want to record a web application but I am not able to record it. The Loadrunner launches the web page but does not record the actions.

Query 2: Hi I have recorded the application using multiple protocols - HTTP and AJAX. But while enhancing the VU Gen script, i am getting some different code snippet against AJAX call used in my web application. Please suggest me how can I overcome the situation.

Here above two queries have been churned out to put across the situation of a beginner and somewhat expert tester. The above two queries need to find out a prominent way of recording a script without any glitch.

Here some blind folded people explain the concept of VUGen Analyzer - "This is the best way to find the protocols." Now there are many cases when you use the VU Gen analyzer and still face the challenges as asked above in Query 2.

Some thumb rules for protocol analysis in VUGEN -

1) Protocol analysis is not a science but an art of understanding the architecture of System Under Test (SUT)
2) Protocol is not about which technology your application is using - For example if your developer has used .NET to develop a web application (aspx pages) does never mean that you will pick .NET protocol for recording a VUGEN script.
3) Most of the web applications work perfectly with HTTP protocol. The only thing you can do is tweak and play your script with some enhancements.
4) AJAX calls can be recorded without even selecting AJAX protocol in VUGen. That's true ! What you need to do is record the application on HTTP protocol and an AJAX transaction will be recorded in custom code. Now that custom code can be used to customize your request.
5) Use .NET protocol when you application type is desktop/windows and is built on C# /VB.NET
6) Use SAP protocol for SAP applications
7) For citrix application use ICA protocol and give proper configuration command while recording the application.

Recording a Loadrunner Script in Virtual User Generator

Recording a Loadrunner Script

Before start recording a loadrunner script, it is vital to have at least below pre-requisites ready -

1) Overview of functionalities of application
2) Business Process ready.
3) Master data generation for playback
4) Analyzing the protocol for your script

I have already covered the above pre-requisites in my previous posts.

I am starting this post with procedure to record a basic HTTP script -

Step 1 -

Click New button OR select File-->New OR simply press ctrl+n keys of your keyboard.
Now you will see the below dialog-

Step 2 -

Since we are recording a web application, so  select Web (HTTP/HTML) protocol. Then you will see the below dialog -

Step 3 -

a) Select Application Type "Internet Application"

b) Program to record "Microsoft Internet Explorer"

c) URL Address "URL of your application"

d) Working Directory "Temporary directory for all LR related tasks"

e) Record into Action "Action" OR click New and then enter any name for e.g. "EnterURL"

     (Please note do not use any special character in giving an action name, also we should give a proper meaningful name to actions.)

Step 4 -

 Click Option button and then select HTML-based script from Recording menu to change the settings of your recording (as given in below image) -

Step 5-

Click HTML Advanced button and make sure to have the following settings and then click OK-

Loadrunner Analysis Graphs

1) Summary page shows the summarized analysis on single page.


2) Running Vusers displays the number of Vusers that executed Vuser scripts, and their status, during each second of a load test. This graph is useful for determining the Vuser load on your server at any given moment.

 3)  Hits per Second displays the number of hits made on the Web server by Vusers during each second of the load test. This graph helps you evaluate the amount of load Vusers generate, in terms of the number of hits.

 4)      Throughput displays the amount of throughput (in bytes) on the Web server during the load test. Throughput represents the amount of data that the Vusers received from the server at any given second. This graph helps you to evaluate the amount of load Vusers generate, in terms of server throughput.

 5)      Transaction Summary displays the number of transactions that passed, failed, stopped, or ended with errors.

6)      Average Transaction Response Time displays the average time taken to perform transactions during each second of the load test. This graph helps you determine whether the performance of the server is within acceptable minimum and maximum transaction performance time ranges defined for your system.

7)      Connections per Second display the number of Connections per Second.

8)      Transaction Response Time Under Load displays average transaction response times relative to the number of Vusers running at any given point during the load test. This graph helps you view the general impact of Vuser load on performance time and is most useful when analyzing a load test which is run with a gradual load.

9)      Transaction Response Time (Percentile) displays the percentage of transactions that were performed within a given time range. This graph helps you determine the percentage of transactions that meet the performance criteria defined for your system.

Performance Testing Process

Performance Testing Process

The performance testing of any application can be carried out based on the NFRs (Non-Functional Requirements). This is normally done with a discussion/meeting with the client/Business Analysts during the project planning phase. Generally this is given in terms of SLA - page response time or transaction response time, Hits Per Second, Throughput etc.




Planning a Performance Test

The following needs to be answered by a client/BA during planning of performance test –

1) Business Processes to be tested

2) Anticipated user load of application

3) Division of user load among business process (based on current logs)

4) Architecture of application

5) Environment for performance testing



For example –In mentioned point 1, three business processes need to be recorded for performance testing – Business Process 1, Business Process 2 and Business Process 3, in point 2, the evaluated application load is 200 virtual users and percentage of each business process is 40%, 30% and 20% respectively- i.e. the actual virtual user load for these business processes will be 80, 60 and 40 respectively.



Designing Performance Test Scenarios

Once the planning of performance testing is completed, performance testing team designs the test cases based on the business process provided by client/BA. These will include the step by step functionality with expected result, and the data requirement for each page. This will help the performance tester to automate the script and will help in anticipating the master/external data requirement for automated script.



Scripting Performance Test Scenarios

After completion of test case designing the performance tester will record the business processes using testing tool (Loadrunner) followed by script enhancements. These scripts will be reviewed by a technical lead/performance architect.




Test Data Generation

The testing/dev team will generate the data for test execution.



Test Scenario Design using Loadrunner Controller

Performance tester will design the scenario/mix for scheduling the groups and configuring monitors. For configuring Solaris counters the testing team can use two approaches –

1) UNIX monitor configuration using rstatd daemon utility (Please refer to Solaris counters )

2) Windows monitor configuration using perfmon (Please refer to Windows Counters)



Test Execution and Monitoring using Loadrunner Controller

After designing all test scenarios the tester will execute the test run and will be monitoring the graphs e.g – Server resource graphs, Running Vusers, Hits per Second, Throughput, Transaction Response Time, Connections per Second etc. Also the server logs can be monitored during test execution.



Test Result analysis using Loadrunner Analyzer

The result will be collated after test execution and using LR interactive graphs, the report can be published in HTML and/or Excel and/or Word document. This report will contain the summary of test and graphs as per the requirement. (Please refer to Analyzer Graphs section)

VU Gen (Virtual User generator) Cond...

FAQs

In order to record a script in VU Gen one may ask few questions about scripting. I am mentioning below some of most frequently asked questions in VU Gen script recording -

Question 1) What is a VUGen script?
 Answer   :   A VUGen script is end-to-end functionality (business flow) of an application, recorded by Loadrunner's component VUGen. Some people call it a "session" in other tools.
Question 2) Which language is used in script recording in LR?
Answer   : VUGen script is recorded in C Language.

Question 3) What is a length of a script?
Answer : Tength of a VUGen script varies from application-to-application, and business process length.

Question 4) What is the extension of a VUGen script file?
Answer : .usr ("dot usr")

Question 5) What is important in VU Gen scripting?
Answer : The flow that you are recording in VUGen is important. Because in some cases it has been observed that the client refuses to accept the result of a performance test stating- this is not what we expected to test. So planning of business process scenarios (flows or functionalities) is the key element of VUGen script recording.

Question 6) What is the cost of VUGen?
Answer:  It's Free. As the VU Gen is a component of Loadrunner that comes with the evaluation version and that too free. The license key is required for the use of Load controller. However, Load controller comes with an evaluation version for 10 days.

Question 7) Is VUGen scripting simple or difficult.
Answer: It is not simple until you initiate it and it is not difficult if you do it.

Question 8) What is correlation in Loadrunner and why correlation is important?
Answer: Correlation is used to capture dynamic values returned from the server. Every iteration requires
              some unique dynamic values that can only be captured using correlation.

Question 9) What are run-time settings in VUGen?
Answer : Run-time settings are configuration settings that to be done before running a script but for during
               run. These settings are simulation settings for a script. For example - You can simulate the
              browsers that are not available on your client machine. You can simulate number of itereations,
              network bandwidth etc.

Question 10) While running the VUGen script, I am not able to see the browser. How can I see that, and what term is used to call that browser?
Answer : The browser that can be watched during every step of your script, is called as "run-time 
                browser". By default it is disabled after fresh installation of VUGen.
               To enable it go to - Tools -- > General Options-- > Display Tab -- >
               Check "Browser during display"--  > Click OK button
               See the below image for quick glimpse.