Visual Studio 2017 Load Testing of an Azure Web App with Application Insights Part 2 of 2

In continuation from my blog post Visual Studio 2017 Load Testing of an Azure Web App with Application Insights Part 1 of 2 , I have shown the web app and azure environment setup that is prepared for a load test of 10,000 simulated users. In this post, I will show the execution, some of the observed metrics and share my opinions, tips and challenges.

The Visual Studio Load Test Result

loadtest7

If you click on the Detail button in the tool bar, you can see the step load pattern of virtual users. Over the course of 15 minutes, I had virtual users run through any of the 3 recorded webtests. As I scroll to the end of the test, I can identify a virtual user of id 9998. Therefore, this shows that I have very closely achieved my 10,000 user test load target.

loadtest8

Azure App Service

The overview blade shows monitoring metrics.loadtest9

Azure Application Insights

Live Metrics Stream

The following shows real time monitoring within the last 60 second window.
Request rate hitting upwards of 100/s and some request duration taking around 2.5 seconds.
loadtest12

Process CPU averaging ~50%
loadtest13

Performance Overview
loadtest14

Server
loadtest11

While the load test was executing, I turned to my browser to manually test responsiveness as I click into the UI and make edits. The responsiveness was slow to 5-7 seconds.

Azure SQL – Performance Overview

Max CPU consumption close to 100%. Green box highlight is the duration of the load test.
loadtest15

A really neat feature of Azure SQL is the ability to review and accept action of performance recommendations. The following shows two indexes being created and another two being validated for creation. Creating indexes will help with query performance.
loadtest16

App Service Plan

Auto Scale

The following shows the auto scale rules being enforced to increase the instance count due to a load test and then decrease back to 1 instance when CPU average < 5% over a 15 min period.
loadtest17

My thoughts and opinion on my load test scenario and results show that an app service plan and azure SQL DB can take decent load for a relatively small traffic site of 10,000 users, but will be noticeably slow for end users. Therefore, in this situation consider scaling out to more 2 more instances and/or scale up to a larger compute size. Also increase the DTU of the Azure SQL DB to perhaps 20 DTU.

Tips and suggestions collected from this load testing experience:

  1. Azure SQL DB Performance Issue
    When the Azure SQL DB was set at Basic pricing tier at 5 DTU, I had the following message in Live Metrics Stream telemetry

    Time     9:32:38 PM
    Duration             10.46 sec
    Outgoing Command       tcp:ABC-cc.database.windows.net,1433 | ABC | Open
    Result code        40613
    ErrorMessage    Database 'ABC' on server 'ABC-cc' is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of '7AB82EFB-53D7-4EB8-A875-81D06BFBEE06'.

    To resolve, I set the pricing tier to Standard at 10 DTU and executed 3 more load tests and was not able to reproduce this issue.

  1. VS Load test system out of memory exception.
    Upon load test completion, one of the errors you may see is an Out of memory exception.
    loadtest18
    The VS Load test application is 32-bit. And so even though I used a VM with 32 GB of RAM, the memory usage is limited since it is using a 32 bit address space.For details and background read https://blogs.msdn.microsoft.com/mahipalkante/2010/06/10/visual-studio-2010-load-test-out-of-memory-exceptions-on-32-bit-machines-or-32-bit-run/
  1. No metrics from load test into App Insights Browser monitoringIn App Insights Browser blade, if you don’t notice any metrics from your VS load test, that is because the simulation of the tests are recorded http requests and don’t involve the app insights JavaScript code running which would be in the client-side browser.
  2. Pushing towards max load users of  >15,000
    In my load testing environment of 8GB RAM, I seemed to manage 10,000 users but ran into various issues when trying to push around 15,000 and more. Careful planning of the load test, web test, Visual Studio settings and the hardware specs are needed to push for much more load of virtual users.

 

  1. Regarding multiple Azure VMs to distribute load tests
    According this article Distributing Load Test Runs Across Multiple Test Machines Using Test Controllers and Test Agents,
    “Test Controllers and Test Agents and not supported on Azure Virtual Machines. To perform distributed load testing for Azure VMs, use the cloud-based load testing service.” 
  2. Cloud-based testing is limited to 20,000 virtual user minutes for free
    Through your visual studio team services (or visual studio online), you can run these cloud-based load tests. There is no option through Azure. I find that the free option is very little for high load you will use the free capacity very quickly. And so I opted for doing my load test on virtual machines.
    For details read:
    https://docs.microsoft.com/en-us/vsts/billing/buy-load-testing-vs#_buy-load-testing
    https://www.visualstudio.com/team-services/cloud-load-testing/
    https://docs.microsoft.com/en-us/vsts/load-test/get-started-simple-cloud-load-test

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s