This article is the second one of a series of 3 where I write about the Cloud Load Testing with Visual Studio Online.
In
the first part I wrote about simple load tests execution using the VSO portal.
In this article I will approach a more complex (but more complete too) solution, the integration between Visual Studio and Visual Studio Online.
Remainder
Before to start with the Cloud Load Test execution, it's important to remind a couple of aspects:
- To use the Cloud Load Test functionality with VSO you need a MSDN Ultimate subscription
- The application you want to test has to be public exposed on the web (because the load test agents are in the cloud and need to reach our app)
- You can use a max of 20.000 free minutes of load test per month. They are "virtual user minutes", which means thet, i.e., if you execute a 2 minutes test with a constant load of 200 virtual users, you're going to consume 400 minutes..
Also, the Load test feature is available only on Visual Studio Ultimate/Enterprise.
That said, let's see how to setup and execute Load Tests with Visual Studio and Visual Studio Online.
Introduction
If we do a Load test with Visual Studio, we will have a plenty of settings we can configure, so this is the most tunable and customizable way to do such kind of tests.
Also, after the execution we will find a lot of information, data and charts about how our application performs under load.
Last but not least, we can do a multi step (multi page) test, even with pages under authentication, and we can record all the steps we want to do in the app in the same way a user will use it (so, no manual configuration is required for this task).
The Start
First of all, we have to create a test project in Visual Studio. The project template to use is "Web Performance and Load Test Project"
When we click on the "OK" button, VS creates for us the solution (as ever...) with a "WebTest1.webtest" project in it. This is not a "Cloud Test" project, it's a "Web Performance" one. We need this kind of project to instruct the load test in what to do.
In the upper left of the project, indeed, there is a small button with a big red dot: this is the button that allow us to record the "steps" in our application.
If we click on this button, it will open a browser session (typically IE) with a small panel on the left. This is the pane that "do the work", it's generated by the "Web Test Recorder helper" that VS installs on your machine.
As side note, remember to check if the plugin is enabled on the browser, if it isn't nothing will show.
Ok, now that the recording is active we can just visit and use the application we want to test just like a normal user will use it. We can also do some specific steps if we need to test specific pages, sections and so on.
To start the recording, write in the browser bar the url of the application and use it, the web recorder plugin will do all by itself.
When we have finished, just click on the "Stop" button in the recording pane. If we don't want to record some step, we can pause the recording and resume it later.
While we navigate our application, you will notice that a lot of "stuff" is added to the recording pane. Those are all the requests our app does.
If we select one of them and then click on the "comment" button, we can set a comment on it to be used later.
When we click on the "Stop" button, some magic happens. Firstly, Visual Studio grabs all the requests from the web recorder plugin, elaborates them and shows them in the webtest window. Then it tries to detect if we have some "dynamic parameters" in our test.
Dynamic parameters are the values that can change between one request and another one made in different moments: user tokens, user search, authentication, etc. If VS detect them, we will be prompted to decide what to do. We can decide to use the ones we recorded for every test it will do or if associate the params with "dictionary" where VS will pick different values at different times.
Another interesting aspect we can customize is related to "Request details". In this window we can decide if we want to insert a "think time" between requests (a delay) and, most important, what is for us the target time for the response. If during the test one request will have a response time greater than what we set, VS will consider the test as failed.
Now we have the webtest with all the steps and settings, we can add the Load test project.
The Load Test project
To add a load test to a web performance test, just right-click on project name, chose "Add" and than "Load test".
This action will start the "Load test wizard" which guides us through the load test setup.
On the first step, we have to select how to manage think times (delay between requests): we can use the "recorded times" (the ones that happens during registration), normal distribution (an average based on recorded times) or to remove all the think times. We can also insert a think time between iterations of the test.
In the second step we have to decide how much load we want to generate, and how to generate it.
We can select to have a constant load (the test will start with N users and will continue with the same number of users until the end) or to have a "step load".
In the image, for example, I decided to start the test with 10 users and then to add other 10 users every 10 second until a maximum of 200 users.
Step load can be really interesting because in that way we will discover how our application scales based on the incremental load.
In the next step, we have to instruct the load test in what to do. For that reason, we will add the web performance test we created before.
Click to "Add", select the performance test and click to "Ok". The test will be added to the "test mix" window. It's called "Test Mix" because we can add as many test as we want, and we can decide the distribution of each test over the total. In this example we have only one test so the distribution will be 100%.
We can also decide, in the next steps, how to mix network types for source users (warning: when I wrote this article VSO supports only "LAN" network type) and also the kind of browser engines we want the test agents use to do the requests.
As you can see in the image, we can select from a list of different browser engines and we can also set the distribution between the user agents.
In the last step, we can customize the load test duration.
We can select between fixed time duration or number of test iterations to be done regardless of the time. If we select a fixed time (2 minutes in my example), we can also insert a "Warm-up time" (10 seconds in the example). During this time some dummy requests will be sent against the application and no result data will be collected. This is extremely useful if we has to test an application that could have a "cold start" (think for example an application hosted on IIS that could not be used for a while and than IIS "deactivated" the app pool) and to not have some false-positives in the test result.
Clicking on "Finish", Visual Studio will generate the load test project and saves it in our solution.
Test execution
Now that we have all the stuff ready, we can start our test execution. But before, only a little more customization. Indeed, we want to execute our load test using the Visual Studio Online Cloud Load test feature, but noone has said this to our local Load Test project.
To do it, open (double click) the "Local.testsettings" file in the solution and change the "Test run location" to Visual Studio Online.
Note that if you have logged in to Visual Studio with an account that own a VSO subscription, this step is not necessary because VS already selected it for you.
Ok, let's start the test. Click on the "Start button" in the upper left.
But what happens when you start the test?
Visual Studio connects to Visual Studio Online and queues the test. VSO then creates an on demand virtual lab in some Azure datacenter and configures the test agents on the VMs, with the given settings.
As soon as the lab is ready and configured, the test starts. We have configured a warm up period, so it starts with it and we can see no data is collected.
Then, the agents start to generate traffic to the given pages of our application (as used in the recording) and the results are posted back to Visual Studio in near real time. In this way we can have a preview of the test outcome.
Now we can wait the test execution ends to have the full resultset.
Results
When the test completes, we are given the results and the achieved performances. We have a lot of information available.
We have the average responses time, with also the min and the max, together with average test times, average page times and user load.
We can also see from the chart and from the info that the virtual users grew from 10 to 120 (I set 200 as maximum but it didn't have time to reach it 'cause the test lasted only 2 minutes) and how the tested application scaled.
Also in the "Throughput" page we can see other useful informations.
In this example we don't have errors, but if they had occurred we'll find them with the generic cause, the specific type and the given error message.
But this is only a small subset of all the information we can have. If we click on the "download report" link, in the upper, VS will download the whole resultset from Visual Studio Online and we can enjoy a sheet plenty of data and charts which can help us to better understand how our application performs under certain load.
Conclusions
This kind of Load Test is the most complete we can do. As we have seen, it gives us a lot of informations, metrics and charts to better understand our application behaviours.
Moreover, the setup of these tests is simple but very customizable and in a while we can have a huge number of useful info.
Stay tuned for the 3rd and last article of this series, which will be about Load Test APIs.
